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ABSTRACT 


This  study  was  concerned  with  the  development  and 
operationalizing  of  a  conceptual  scheme  for  formative 
evaluation  of  a  simulation  called  Transition .  Transition 
was  designed  as  part  of  a  curriculum  package  of  Project  SURT 
(Study  of  Urban  Rural  Transition) ,  a  subproject  of  Project 
Canada  West,  for  use  by  students  of  Canadian  secondary  school 
social  studies. 

As  an  integral  part  of  the  developmental  process,  the 
formative  evaluation  was  considered  a  formalized  evaluation 
of  Transition  to  identify  deficiencies  and  successes  to 
enable  the  making  of  revisions  in  its  formative  stage  of 
development.  The  intent  of  the  study  was  therefore  to 
provide  feedback  information  into  the  developmental  process 
so  that  decision-makers,  the  simulation  development  team, 
could  make  improvements  in  their  product. 

Three  basic  components  of  Transition  were  recognized 
as  being  worthy  of  evaluation:  the  sequence  of  activities, 
the  scenario  of  materials  and  the  simulation  model.  The 
model  for  Transition  was  viewed  as  having  two  dimensions — a 
substantive  dimension  as  presented  by  the  Community  Systems 
Model,  and  a  process  dimension  as  presented  by  the  method  of 
inquiry  postulated  by  Charles  W.  Morris  and  incorporated  into 
the  decision-making  process  of  the  simulation. 

The  evaluation  of  Transition  was  conducted  in  several 
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phases.  A  preactive  phase  occupied  the  investigator  with 
questions  of  what  and  how  to  evaluate.  An  ethnographic 
investigation  of  the  simulation  process  rather  than  an 
experimental  study  of  a  pretest-posttest  nature  was  chosen 
as  the  means  of  formative  evaluation.  Data  sources  which 
facilitated  the  description  and  judgement  of  participant 
transactions  with  the  simulation  were  observations  of  the 
simulation  leader  and  non-participants,  audiotapes  and  video¬ 
tapes  of  selected  segments  of  the  simulation  experience, 
participant  interviews,  a  participant  questionnaire  and 
classroom  documents  including  participant  worksheets  and 
speeches . 

The  first  interactive  phase  involved  an  initial 
testing  with  volunteer  grade  nine  students  at  Vernon  Barford 
Junior  High  School.  The  second  interactive  phase  consisted 
of  a  major  piloting  with  a  grade  nine  class.  Their  simulation 
experience  occupied  seven  hours  of  actual  class  time  over  four 
teaching  days.  Both  initial  testing  and  major  piloting  were 
observed  in  their  entirety  by  the  investigator. 

Assessment  and  revisions  of  the  simulation  were  made 
in  two  stages.  In  the  first  postactive  phase  data  were 
analyzed  from  an  ethnographic  perspective.  The  second  post¬ 
active  phase  involved  the  development  team  in  an  extensive 
assessment  of  the  simulation  model  and  revision  of  all 
materials  and  activities. 

This  study  has  clearly  demonstrated  that 
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investigation  into  the  process  dimension  of  a  simulation  in 
its  developmental  stage  by  means  of  an  ethnographic  research 
approach  is  an  appropriate  means  of  formative  evaluation. 
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CHAPTER  1 


INTRODUCTION 

1.  THE  PURPOSE  OF  THE  STUDY 

The  purpose  of  this  study  was  to  describe  and 
evaluate  the  transaction  between  the  participant  students  and 
the  simulation  called  Transition  as  a  means  of  revision  of 
the  simulation.  Transition  was  constructed  to  simulate  the 
problems  faced  by  Canadian  families  as  they  move  from 
rural  to  urban  and  urban  to  rural  centres.  The 
simulation  was  developed  as  part  of  the  curriculum  package 
of  Project  SURT,  a  subproject  of  Project  Canada  West,  and 
was  designed  for  students  of  secondary  school  social  studies. 
In  this  study  the  investigator  attempted  to  determine  the 
variance  between  "what  was"  and  "what  was  desired"  when 
students  transacted  with  displays  contained  within  the 
simulation  so  as  to  enable  the  development  team  to  make 
necessary  revisions  in  the  first  draft  of  their  product. 

11.  THE  PROBLEM 

The  problem  under  investigation  was  how  to  conduct 
a  formalized  formative  evaluation  of  three  basic  components 
of  Transition:  the  simulation  model,  the  sequence  of  activi¬ 
ties  and  the  scenario  of  materials.  As  a  means  of  formative 
evaluation,  the  study  attempted  to  answer  the  following 
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questions : 

1.  What  inquiry  process  do  participants  use  in  handling 
information  presented  them  by  the  simulation  and  how 
well  do  they  use  the  structured  inquiry  process  of 
the  simulation? 

2.  What  activities  are  participants  engaged  in  during 
their  simulation  experience  and  how  well  do  they 
participate  in  the  identified  activities  of  the 
simulation? 

3.  What  are  the  participants'  perceptions  of  and 
attitudes  toward  their  experience  of  participation 
in  Transition? 

111.  THE  NEED  FOR  THE  STUDY 

Formative  evaluation  is  vital  in  the  preactive 
phase  of  instruction,  the  development  of  instructional  plans. 
The  process  of  piloting,  debugging  and  revision  must  take 
place  during  the  formulation  of  intended  outcomes,  design  of 
instrumental  content  and  development  of  teaching  strategies. 
Evaluation  of  these  components  of  an  instructional  plan  should 
not  be  treated  casually  but  rather  should  be  carefully  built 
in  as  part  of  the  developmental  process. 

This  study  was  undertaken  to  serve  the  immediate  need 
of  formative  evaluation  of  Transition  in  which  have  been 
incorporated  numerous  intended  outcomes,  strategies  and 
displays.  As  well,  the  study  was  conducted  to  test  the 
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importance  of  systematic  evaluation  during  the  developmental 
process.  Instructional  planners  tend  to  treat  evaluation 
very  casually,  often  as  an  afterthought.  This  formalized 
formative  evaluation  was  conducted  based  on  the  belief  that 
evaluation  is  crucial  in  the  developmental  process. 

IV.  DEFINITION  OF  TERMS 

In  this  study  the  following  definitions  of  terms 
were  used. 

Formative  Evaluation 

Formative  evaluation  is  a  role  of  evaluation  as 
distinct  from  summative  evaluation,  both  of  which  operate  at 
different  times  in  the  formulation  of  outcomes  and  develop¬ 
ment  of  instrumental  content  and  teaching  strategies.  It  is 
the  feedback  of  product  information  into  the  developmental 
process  while  it  is  still  fluid.  (Scriven,  1967) 

Instructional  Planning 

Instructional  planning  is  the  preactive  phase  of 
teaching,  the  phase  which  links  curriculum  and  instruction. 
Instructional  planning  has  two  major  components:  (1)  instruc¬ 
tional  content,  and  (2)  teaching  strategies.  The  instructional 
content  component  consists  of  two  sub-components:  (1)  the 
intended  outcomes  and  (2)  instrumental  content.  (Johnson, 

1968;  Aoki,  1970) 
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Instrumental  Content 

Instrumental  content  is  that  content  which  is 
instrumental  to  the  goal,  the  intended  outcome.  It  is  the 
object,  event  or  action  with  which  students  transact 
cognitively,  affectively  and  conatively.  Instrumental 
content  is  a  major  component  of  an  instructional  plan. 
(Johnson,  1968;  Aoki,  1970) 

Simulation 

Simulation  is  a  symbolic  model  of  a  social  situation. 
It  is  a  selective  representation  of  reality  containing  those 
elements  of  reality  deemed  relevant  by  the  designer.  Its 
purpose  is  to  provide  situations  in  which  participants  can 
experience  decision-making.  (Coleman,  1968) 

Educational  Simulation* 

Educational  simulation  is  a  plan  for  instruction 
which,  when  operationalized,  allows  students  to  assume  the 
role  of  decision-makers  in  a  simulated  environment  which  has 
specified  rules  and  procedures.  It  is  designed  to  give 
students  a  vicarious  experience  of  a  social  process  which 
they  may  meet  in  later  life. 

Decision-making 

Decision-making  is  a  fundamental  cognitive  and 
affective  activity  of  man.  It  is  a  process  whereby  an 
individual  chooses  one  of  a  number  of  alternatives  in  both 
*Henceforth  referred  to  as  simulation. 
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appraisive  and  prescriptive  situations;  i.e.,  when  he  sets 
goals  and  determines  actions  to  be  taken  to  achieve  goals. 

It  may  be  a  rational  and/or  non-rational  process. 

Variance 

Variance  is  a  lack  of  congruence  between  "what  is" 
and  "what  should  be"  in  an  instructional  situation. 

Inquiry 

Inquiry  is  a  reflective  process  involving  signs  and 
directed  to  solving  problems  so  that  action  may  proceed.  It 
may  be  of  three  types.  Designative  inquiry  deals  with 
problems  of  what  has  happened,  is  happening  or  will  happen. 
Appraisive  inquiry  deals  with  problems  of  what  should  be. 
Prescriptive  inquiry  deals  with  problems  of  what  should  be 
done.  The  three  modes  of  inquiry  are  complexly  interrelated 
and  never  completely  separated.  Any  inquiry  process  can  have 
a  designative,  appraisive  or  prescriptive  configuration. 
(Morris,  1964;  Aoki,  1970) 

Transaction 

Transaction  is  the  fundamental  interaction  in  instruc¬ 
tion  between  student  and  selected  segments  of  his  environment, 
that  is,  the  instrumental  content  displayed.  In  the  classroom 
setting  the  fundamental  interaction  is  complicated  by  the 
mediating  presence  of  an  instructor  and  other  students. 

There  is  interpersonal  interaction  which  has  a  large  bearing 
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on  the  affective  climate  of  the  social  setting.  (Johnson, 

1968;  Aoki ,  1970) 

Ethnography 

Ethnography  is  a  method  of  describing  a  particular 
culture  from  a  first-hand  investigation.  Culture  is 
restricted  to  mean  the  knowledge  people  use  to  generate  and 
interpret  social  behavior.  The  ethnographer  selects  informants 
from  a  cultural  scene.  These  informants  share  information 
that  defines  some  aspect  of  their  experience  in  a  social 
situation.  They  in  fact  teach  the  ethnographer  in  the  ways 
of  life  which  they  find  meaningful.  (Spradley  and  McCurdy, 
1972) 


V.  THE  SIGNIFICANCE  OF  THE  STUDY 

This  study  emerged  in  part  from  an  interest  in  a 
segment  of  the  Curriculum  and  Instruction  Design  Model 
developed  by  Johnson  (1968)  and  adapted  by  Aoki  (1970).  The 
segment  of  special  concern  was  instruction  during  which 
students  transact  with  selected  segments  of  their  environment 
(displays) .  In  this  systematic  evaluation,  Transition  was 
considered  as  a  general  display  within  which  there  were  many 
displays  with  which  students  could  transact.  The  investigator 
feels  that  this  study  has  contributed  to  an  illumination  of 
the  transaction  process  in  the  classroom. 

This  formalized  evaluation  study  assisted  the 
developers  in  making  further  revisions  after  the  first  major 
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piloting.  Viewed  as  an  initial  step  in  the  investigation  of 
a  simulation  process,  this  study  contributed  to  an  under¬ 
standing  of  what  goes  on  during  a  simulation  experience  from 
the  viewpoints  of  participants  and  observers. 

Many  simulation  studies  have  placed  great  emphasis  on 
objectivity,  quantification  and  experimentation.  This  study 
has  shown  ethnography  to  be  a  promising  alternative  to 
statistical  research  and  an  approach  which  fits  well  into 
formative  evaluation  of  curricular  materials.  The  investigator 
feels  that  this  study  is  significant  in  demonstrating  that  a 
description  and  assessment  of  the  process  dimension  of  a 
simulation  is  a  logical  and  worthwhile  focus  for  simulation 
investigations  at  least  in  the  formative  stage  of  development. 

VI.  SCOPE  AND  LIMITATIONS 

This  study  was  limited  to  a  formative  evaluation  of 
a  specific  simulation  in  the  development  of  which  the 
investigator  was  directly  involved.  The  evaluation  was 
specifically  intended  to  provide  feedback  information  into 
the  developmental  process  for  the  purpose  of  improving  the 
simulation. 

In  essence  the  study  was  both  descriptive  and 
judgemental.  It  attempted  to  describe  and  assess  the 
transactions  between  participant  students  and  the  simulation 
and  to  determine  student  perceptions  of  and  attitudes  to 
their  simulation  experience.  No  attempt  was  made  to  determine 


f  ■ 


■ 


-■  - 

. 


. 


8 


the  learning  effects  of  the  simulation  or  the  effects  on  the 
decision-making  ability  of  participants.  The  study  can  be 
viewed  as  an  initial  investigation  into  the  process  dimension 
of  a  simulation. 

One  grade  nine  Alberta  Social  Studies  class  of  28 
students  was  involved  in  the  major  class  piloting.  This  major 
piloting  was  confined  to  a  duration  of  seven  hours  over  four 
teaching  days;  i.e.,  three  consecutive  days  and  one  five  days 
later.  An  initial  testing,  confined  to  one  four  hour  session, 
with  11  volunteer  grade  nine  students  not  involved  in  the 
major  class  piloting  was  conducted  prior  to  the  major 
piloting . 

An  obvious  limitation  was  the  Hawthorne  effect.  The 
major  piloting  was  a  novel  situation  for  students  in  that  the 
setting  was  the  school  library  rather  than  the  regular 
classroom.  Two  videotape  recorders  and  four  audio  recorders 
were  in  operation  during  selected  segments  of  the  simulation 
experience.  Effort  was  made  to  control  for  the  Hawthorne 
effect  however  by  restricting  the  number  of  outsiders  to 
three,  two  cameramen  and  the  investigator  himself,  and  by 
positioning  the  recording  equipment  so  as  to  be  as  incon¬ 
spicuous  as  possible. 

VIII.  PLAN  OF  THE  REPORT 

A  review  of  literature  pertaining  to  formative 
evaluation,  trends  in  simulation  evaluations  and  the 
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ethnographic  approach  to  evaluation  is  reported  in  Chapter 
2.  In  Chapter  3  are  presented  the  rationale,  framework  and 
actual  procedures  for  the  development  of  the  simulation 
Transition.  Chapter  4  deals  with  the  manner  in  which  the 
conceptual  scheme  for  formative  evaluation  became  operation¬ 
alized.  Major  revisions  of  the  simulation  activities  and 
an  assessment  of  the  simulation  model  are  presented  in 
Chapter  5.  In  Chapter  6  are  presented  a  summary  and 
conclusion  of  the  study  as  well  as  a  number  of  recommendations 
for  further  research. 
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CHAPTER  2 


REVIEW  OF  RELATED  LITERATURE 

In  this  chapter  is  reviewed  literature  pertaining  to 
the  following  major  themes: 

1.  Formative  Evaluation:  A  Conceptual  Scheme. 

2.  Evaluation  of  Simulations:  Past  and  Present  Trends. 

3.  Ethnography:  A  Conceptual  Scheme. 

FORMATIVE  EVALUATION:  A  CONCEPTUAL  SCHEME 

This  section  is  concerned  with  a  review  of  literature 
related  to  evaluation  in  general  and  formative  evaluation  in 
particular.  The  literature  related  specifically  to  the  goals 
and  roles  of  formative  evaluation  has  provided  the  investigator 
with  a  conceptual  base  for  the  evaluation  of  Transition  in 
its  formative  stage  of  development. 

In  an  overview  of  readings  in  curriculum  evaluation, 
Taylor  and  Cowley  (1972)  pointed  out  that  there  are  two 
dimensions  to  evaluation:  a  descriptive  dimension  and  a 
judgemental  dimension.  According  to  them,  the  more  critical 
and  complex  judgemental  dimension  has  often  been  overlooked 
by  evaluators.  Taylor  and  Cowley  describe  the  most  recent 
post-Sputnik  era  in  evaluation  as  characterized  by  the 
realization  that  the  domain  of  evaluation  must  be  extended  to 
incorporate  the  all  important  judgemental  dimension. 

Cronbach  (1963)  defined  evaluation  broadly  as 
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"the  collection  and  use  of  information  to  make  decisions 
about  an  educational  program"  (p.  672) .  He  emphasized  that 
the  greatest  service  evaluation  can  perform  is  to  identify 
aspects  of  a  course  where  revision  is  desirable.  In  Cronbach's 
words : 

Evaluation  used  to  improve  a  course  while  it  is 
still  fluid  contributes  more  to  the  improvement  of 
education  than  evaluation  used  to  appraise  a  product 
already  placed  on  the  market.  (1963;  p.  677) 

As  Cronbach  has  indicated,  evaluation  must  occur  during  the 

preactive  or  planning  phase  of  instruction  rather  than 

solely  afterwards  when  developers  are  more  reluctant  to 

revise  the  intended  outcomes,  instrumental  content  and 

strategies  incorporated  in  their  products. 

Similar  views  on  evaluation  have  been  voiced  more 
recently.  Scriven  (1967)  pointed  out  that  evaluation  should 
be  a  fundamental  part  of  curriculum  development,  not  an 
appendage  or  afterthought.  He  has  criticized  evaluation 
research  which  assumes  that  one's  obligations  in  the  direction 
of  evaluation  are  fully  discharged  by  having  it  appear 
somewhere  in  a  project. 

In  his  formulation  of  a  conceptual  framework  for 
curriculum  evaluation,  Scriven  differentiated  between  the 
goal  of  evaluation  and  the  role  of  evaluation.  The  goal  of 
evaluation  is  the  answering  of  product-oriented  questions 
about  educational  instruments;  whereas  the  role  of  evaluation 
refers  to  the  use  to  which  evaluation  is  to  be  put.  Scriven 
further  distinguished  two  roles  of  evaluation,  formative  and 
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summative,  which  operate  at  different  times  in  the  develop¬ 
ment  of  programs  and  more  specific  products. 

According  to  Scriven,  formative  evaluation  refers  to 
the  feedback  of  product  information  into  the  developmental 
process  for  the  purpose  of  improving  the  product.  Summative 
or  terminal  evaluation  involves  the  collection  of  data 
concerning  an  already  operating  program  for  the  purpose  of 
making  judgements  about  the  program.  Scriven  has  emphasized 
that  curriculum  projects  must  try  to  make  best  use  of 
evaluation  in  both  of  these  roles.  In  emphasizing  the  role  of 
formative  evaluation,  Scriven  said: 

Any  curriculum  reformer  is  automatically  engaged 
in  formative  evaluation  ....  As  he  proceeds  to 
construct  the  new  material  he  is  constantly  revising 
his  own  material  as  better  than  that  which  is  already 
current  ....  He  is  presumably  engaged  in  field 
testing  the  work  while  it  is  being  developed  and  in 
so  doing  gets  feedback  on  the  basis  of  which  he  can 
produce  revisions;  this  of  course  is  formative  evalua¬ 
tion.  (1967;  p.  30) 

Both  Cronbach  (1963)  and  Scriven  (1967)  recognized 
the  importance  of  systematic  formative  evaluation  in  the 
discovery  of  "deficiencies  and  successes  in  the  intermediate 
versions  of  a  new  curriculum".  Cronbach  in  particular 
identified  numerous  approaches  to  systematic  evaluation. 

These  include  process  studies,  proficiency  studies,  attitude 
and  follow-up  studies.  Process  studies  are  concerned  with 
events  taking  place  in  the  classroom.  According  to  Cronbach, 
process  measures  such  as  videotapes,  audiotapes  and  direct 
observations  of  classroom  transactions  have  special  value 
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because  they  focus  on  what  happens  during  instruction. 
Proficiency  studies  utilize  standardized  tests  as  the  main 
source  of  feedback  data.  Attitude  studies  usually  take  the 
form  of  direct  and  indirect  questioning  via  interviews  and 
questionnaires  and  can  be  still  another  valuable  source  of 
feedback  into  the  developmental  process.  Cronbach  described 
follow-up  studies  as  being  least  important  for  formative 
evaluation  since  they  are  so  far  removed  from  the  develop¬ 
mental  process. 

The  literature  related  to  curriculum  evaluation 
clearly  indicates  that  the  particular  approach  to  evaluation 
which  developers  use  greatly  affects  the  nature  of  the 
feedback  information  and  therefore  the  entire  revision 
process.  As  Cronbach  has  pointed  out,  the  advantages  and 
disadvantages  of  each  approach  must  be  carefully  weighed 
before  the  intermediate  versions  of  intended  outcomes, 
instrumental  content  and  strategies  incorporated  in  a  product 
are  formatively  evaluated. 

EVALUATION  OF  SIMULATION:  PAST  AND  PRESENT  TRENDS 

Numerous  evaluative  studies  have  been  carried  out 
since  the  first  appearance  of  simulations  on  the  education 
scene.  Although  these  studies  exhibit  great  variation  in 
terms  of  quality,  scope,  design  and  findings,  they  are  similar 
in  their  focus  on  the  impact  of  simulations  on  participants, 
both  cognitively  and  affectively.  Rather  than  enumerate  the 
many  investigations  into  the  impact  of  simulations  which  have 
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been  carried  out,  selected  simulation  literature  and  research 
studies  are  reviewed  to  indicate  the  general  trends  in 
simulation  evaluation. 

The  Meaning  of  Simulation 

The  distinction  between  games  and  simulations  has 
been  a  matter  of  much  dispute  among  social  scientists  and 
educators.  The  terms  gaming,  simulation  and  simulation  games 
have  been  used  synonymously  by  some  writers  while  others  are 
adamant  in  opposing  this  usage.  Those  who  distinguish  between 
"game"  and  "simulation"  equate  the  former  with  a  contest, 
involving  competition  among  participants  and  yielding  a  winner 
and  the  latter  with  co-operation  in  the  achievement  of  common 
goals.  Nesbitt  (1968)  attributed  the  choice  of  terms  to  the 
particular  image  which  educators  are  trying  to  project  rather 
than  the  meaning  of  the  words  themselves.  He  noted  that  the 
word  "simulation"  is  used  with  those  who  might  be  suspicious 
of  using  "frivolous"  games  in  the  classroom  whereas  the  word 
"game"  is  often  used  with  students  who  might  associate 
"simulation"  with  "work". 

Rather  than  providing  an  all-embracing  definition  of 
"game"  and  "simulation",  Nesbitt  investigated  how  games  and 
simulations  were  used  as  educational  tools.  He  defined 
"simulation"  as: 

.  .  .  an  operating  model  of  a  physical  or  social 

process  ...  a  selected  representation  of  reality, 
containing  only  those  elements  of  reality  that  the 
designer  deems  relevant  to  his  purpose.  (1968,  p.  38  ) 
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Nesbitt  pointed  to  competition  for  specified  objectives  as  a 
key  element  in  games  and  concluded  that  the  kind  of  technique 
used  in  most  classrooms  combined  the  elements  of  simulations 
and  games.  Students  assume  the  role  of  decision-makers  in  a 
simulated  environment  and  compete  for  certain  objectives 
according  to  specified  rules  and  procedures.  It  is  for  these 
reasons  that  Nesbitt  and  others  have  preferred  to  use  the 
term  "simulation  game"  to  convey  the  essence  of  this 
technique . 

The  developers  of  Transition  have  preferred  to 
describe  their  product  as  a  simulation.  The  investigator  has 
attempted  to  be  consistent  in  his  use  of  the  term  "simulation" 
in  this  report. 

Research  Dealing  with  Simulations 

One  of  the  first  major  reviews  of  research  dealing 
with  simulations  was  conducted  by  Cherryholmes  (1966) .  She 
examined  the  research  findings  from  the  following  six  simula¬ 
tion  studies:  Anderson  (1964),  Boocock  (1963),  Boocock  and 
Coleman  (1965) ,  Cherryholmes  (1966) ,  Garvey  and  Seiler  (1966) 
and  Robinson,  Anderson,  Herman  and  Snyder  (1966) .  Cherryholmes 
hypothesized  that  simulation  participants  would  reveal  more 
interest,  retain  more  facts  and  principles  of  information, 
retain  information  longer,  acquire  more  critical  thinking  and 
decision-making  skills  and  have  attitudes  more  significantly 
altered  than  students  participating  in  more  conventional 
classroom  activities.  She  concluded  that  simulations  do 
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produce  more  student  motivation  and  interest  but  found  no 
significant  differences  in  learning,  retention,  critical 
thinking  or  attitude  change. 

While  the  quantity  of  simulation  research  expanded 
considerably  between  1966  and  1971,  the  quality  of  available 
research  did  not  particularly  improve.  Fletcher  (1971) 
criticized  simulation  research  for  being  still  in  its  infancy 
stage.  He  cited  the  following  reasons: 

1.  Lack  of  standardized  workable  games. 

2.  Extreme  variations  in  objectives,  structure,  subject 
matter,  task  complexity  and  interaction. 

3.  Inadequacy  of  simulation  models. 

4.  Disagreement  on  appropriate  administrative  procedures 
when  simulations  are  used  for  learning. 

Fletcher  concluded  that  most  simulation  research  was  of  a 
"shotgun"  nature  consisting  of  single  studies  on  particular 
simulations  each  using  its  own  tests  to  measure  different 
sets  of  dependent  and  independent  variables. 

Information  regarding  the  effectiveness  of  simulations 
in  the  classroom  is  limited  in  spite  of  their  popular 
acceptance.  Wentworth  and  Lewis  (1973)  carried  out  a 
systematic  review  of  the  research  conducted  on  instructional 
games  and  simulations  in  social  studies  education.  They 
pointed  out  that  although  most  simulations  are  designed  to 
influence  the  cognitive,  attitudinal  and  behavioral  charac¬ 
teristics  of  participants,  most  of  the  research  on  simulations 
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has  investigated  only  aspects  of  one  or  two  of  these  variables 
They  classified  available  research  into  six  categories 
having  to  do  with  the  investigation  of: 

1.  the  feasibility  of  using  simulations  in  the  classroom, 

2.  personal  characteristics  of  participants  as  variables 
related  to  their  performance, 

3.  student  behaviors,  strategies  and  interactions  during 
participation  in  simulations, 

4.  the  use  of  simulations  to  teach  thinking  skills, 

5.  the  impact  of  simulations  on  other  cognitive  learning, 

6.  the  impact  of  simulations  on  the  interest  and 
attitudes  of  participants. 

In  a  reaffirmation  of  the  conclusions  of  Cherryholmes 
(1966)  and  Fletcher  (1971) ,  Wentworth  and  Lewis  described  most 
simulation  research  as  suspect  because  of  inadequate  testing 
procedures  and  research  design,  use  of  unsophisticated 
statistics  and  the  inability  to  determine  important  variables. 
They  drew  the  following  conclusions  regarding  simulations  as 
teaching  devices: 

1.  Simulations  do  not  appear  to  have  any  clear  advantage 
in  teaching  content. 

2.  Simulations  appear  to  have  a  positive  influence  on 
participant  attitudes. 

Simulations  appear  to  be  influential  in  encouraging 
students  to  become  more  actively  involved  in  the 
learning  process. 
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Available  research  on  simulations  has  been  criticized 
recently  for  probing  questions  of  convenience  rather  than 
those  of  importance  to  users  and  developers.  Many  of  the 
evaluations  of  simulations  have  relied  on  paper  and  pencil 
tests  which  have  limited  measurement  capabilities  for 
simulation  investigations.  In  their  critique  of  evaluation 
procedures,  Wentworth  and  Lewis  indicated  the  extreme 
difficulty  for  the  simulation  process  to  be  measured  by  paper 
and  pencil  tests  given  at  the  end  of  the  simulation  experience. 
In  their  own  words: 

.  .  .  The  weakness  of  paper  and  pencil  tests  is  that 

they  measure  secondary  data  of  a  static  nature.  They 
tell  us  what  students  have  learned.  This  past  tense 
information  tells  us  little  about  how  the  learning 
environment  can  be  manipulated  to  maximize  student 
achievement  and  understanding.  (1973,  p.  439) 

An  analysis  of  the  three  reviews  of  Cherryholmes 
(1966) ,  Fletcher  (1971)  and  Wentworth  and  Lewis  (1973) 
indicates  that  the  great  majority  of  simulation  evaluation 
studies  have  focused  on  the  impact  of  simulations  on  the 
cognitive  and  affective  learning  of  participants.  Remaining 
unanswered  are  questions  such  as  'What  behaviors  are  observed 
when  people  participate  in  simulations?',  'What  modes  of 
inquiry  do  participants  use  at  various  stages  of  their 
simulation  experience?'  These  unanswered  questions  clearly 
indicate  the  need  for  an  investigation  of  the  actual  simulation 
process  as  a  means  of  determining  what  students  do  when  they 
are  learning  rather  than  what  they  have  learned. 
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The  notion  of  investigating  the  simulation  process 
is  receiving  more  attention  in  current  literature  on 
simulations.  In  a  position  paper  on  the  evaluation  of 
simulations,  Instructional  Simulations  Incorporated  (I.S.I., 
1972)  have  distinguished  three  objects  of  simulation  evaluation 
a  simulation  model,  a  sequence  of  operations  and  a  scenario 
of  materials. 

A  system's  model  or  representation  of  a  real  world 
system  is  the  core  of  the  simulation.  According  to  I.S.I., 
it  is  most  important  that  evaluators  consider  whether  the 
simulation  model  has  the  integrity  of  a  system.  Such  a  test 
involves  two  major  questions:  First,  does  the  model  allow  a 
flow  of  actual  data  in  the  manner  which  approximates  the  real 
world  system  being  replicated?  Second,  can  the  model  replicate 
and  respond  to  the  full  range  of  data  that  might  be  presented 
to  it?  I.S.I.  have  emphasized  the  importance  of  the  simulation 
model  having  validity  in  the  system's  sense  and  integrity  if 
students  are  to  learn  a  community  model  through  participation 
in  the  simulation. 

A  second  basis  for  evaluation  lies  in  the  sequence  of 
operations,  the  series  of  tasks  which  correspond  to  the  tasks 
and  functions  of  the  models  being  simulated.  The  sequence  of 
operations  has  been  overlooked  as  simulation  research  has 
instead  focused  on  what  students  have  learned.  I.S.I.  have 
recognized  the  necessity  of  an  investigation  of  process.  If 
students  are  unable  to  perform  a  task,  whether  it  involves 
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negotiating  or  making  a  decision,  then  the  task  components  of 
the  simulation  must  be  revised  and  resequenced. 

A  third  simulation  component  is  the  scenario  which 
allows  various  states  of  the  world  to  be  introduced  to 
participants.  The  scenario  may  or  may  not  be  enriched  with 
background  materials  and  resources.  Nevertheless,  according 
to  I.S.I.,  the  importance  of  a  balanced  scenario  cannot  be 
underestimated  in  evaluative  terms.  Students  must  have  the 
opportunity  for  in-depth  involvement  with  simulation  materials. 

I.S.I.  have  urged  future  evaluators  of  simulations  to 
develop  "process  measures"  rather  than  recall  or  transfer 
learning  indices  or  some  ex  post  facto  design.  Their  elabora¬ 
tion  of  the  components  of  a  simulation  have  indicated  the 
necessity  for  a  simulation  evaluation  to  focus  squarely  on 
the  variables  and  factors  which  prompt  model  integrity  and 
operational  validity. 

There  have  been  few  evaluation  studies  directed  at  an 
investigation  of  participant  transactions  during  their  simula¬ 
tion  experience.  Included  among  the  limited  number  of  process 
studies  which  have  attempted  to  identify  important  behaviors 
during  a  simulation  experience  are  those  of  Feldmiller  (1970) , 
Chartier  (1972)  and  Wentworth  (1972) . 

A  study  by  Feldmiller  (1970)  was  based  on  the  inter¬ 
action  of  participants  in  the  home  economics  simulation 
Choice  and  Chance.  It  was  found  that  the  verbal  behavior  of 


participants  could  be  recorded  and  categorized  so  that  key 
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questions,  words  and  issues  could  be  identified  and  analyzed. 
Feldmiller  discovered  that  participants  tended  to  ask  more 
procedural  and  directional  questions  initially  and  gradually 
became  more  independent  of  the  instructor.  While  the  value 
of  the  findings  was  not  adequately  discussed,  the  study  is 
significant  in  being  one  of  the  first  to  analyze  the  simula¬ 
tion  process  rather  than  measure  student  differences  before 
and  after  the  simulation  or  measure  results  in  terms  of 
stated  goals. 

Chartier  (1972)  used  the  game  Generation  Gap  with 
undergraduate  students  in  an  introductory  speech  class  to 
investigate  the  value  of  linking  discussion  with  simulations 
in  order  to  maximize  their  learning  potential.  He  hypothesized 
that  subjects  who  participated  in  a  simulation  with  a 
discussion  component  would  demonstrate  more  learning  at  the 
cognitive  and  affective  levels  than  subjects  participating  in 
a  simulation  without  discussion,  discussion  without  simulation 
or  neither  (independent  study).  Chartier  administered  a 
predisposition  indicator  to  each  subject  and  developed  an 
achievement  test  to  measure  levels  of  cognitive  learning. 

The  only  significant  difference  between  experimental  and 
control  groups  was  that  subjects  who  participated  in  a 
simulation  with  discussion  demonstrated  higher  learning 
outcomes  at  the  cognitive  level  than  subjects  participa¬ 
ting  in  either  simulation  or  discussion. 

Evidence  supporting  claims  that  simulations  generate 
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more  interest  and  motivation  than  more  conventional  class 
activities  has  been  quite  substantial:  Anderson  (1964), 

Cohen,  Dill,  Kuehn  and  Winters  (1968),  Shirts  and  Sprague 
(1966),  Cohen  (1969).  However,  the  findings  of  these  and 
Chartier's  study  may  be  attributed  to  nothing  more  than  the 
Hawthorne  or  novelty  effect  produced  by  the  newness  of  the 
simulation  method.  Indeed  the  validity  of  all  studies  which 
attempted  to  measure  the  learning  effects  of  simulations 
without  carefully  controlling  for  the  Hawthorne  effect  can  be 
seriously  questioned. 

A  study  by  Wentworth  (1972)  attempted  to  determine 
the  effectiveness  of  the  game  Marketplace  for  teaching 
introductory  economics  in  selected  two  year  colleges.  In 
order  to  measure  the  cognitive  and  affective  learning  effects, 
Wentworth  conducted  an  investigation  of  student  verbal 
behavior  during  their  participation.  He  found  that  student 
responses  became  more  positive,  co-operative  and  reinforcing 
as  the  simulation  progressed.  Student  verbal  behavior  showed 
no  evidence  for  convergent,  divergent  or  evaluative  thinking. 
There  were  no  additional  gains  in  positive  attitudes  toward 
economics  and  little  evidence  to  indicate  carryover  into 
subsequent  classroom  interactions. 

Wentworth's  study  is  significant  in  its  attempt  to 
determine  the  broad  impact  of  a  simulation  by  the  measure¬ 
ment  of  attitudinal,  cognitive  and  behavioral  variables 
during  simulation  participation.  The  study  was  concerned  with 
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the  simulation  process  but  merely  as  a  means  of  determining 
the  learning  effects  of  the  simulation.  No  attempt  was  made 
to  investigate  the  simulation  process  per  se. 

In  an  enumeration  of  the  many  research  problems 
associated  with  gaming  and  simulation,  Goodman  (1973) 
postulated  that  the  fact  that  learning  related  to  games  takes 
place  both  during  and  after  the  game  experience  has  served  to 
obscure  the  problem  of  doing  research  on  games.  Much  of  the 
simulation  literature  refers  to  the  importance  of  post-game 
debriefing  discussions.  However,  according  to  Goodman,  there 
is  reason  to  believe  that  reflections  on  this  debriefing 
process  do  not  terminate  when  a  game  terminates. 

The  problem  for  simulation  research,  in  the  words  of 
Goodman,  is: 


to  develop  a  broad  enough  research  paradigm  to 
permit  the  analysis  of  the  way  games  infect  a  total 
learning  environment.  (1973,  p.  934) 

There  is  a  need  for  simulation  researchers  to  focus  on  the 

formalized  nature  of  simulation  exercises  so  as  to  produce 

data  which  might  in  some  way  indicate  what  is  actually  going 

on  as  participants  engage  in  decision-making.  There  is  an 

appropriate  posture  for  simulation  research  to  assume. 

Goodman  has  described  it  in  the  following  way: 

It  would  appear  far  more  exciting  and  profitable  to 
allow  efforts  to  determine  when  (under  what  circumstances) 
gaming  "works"  with  efforts  to  discover  why  it  works 
rather  than  to  move  toward  producing  ever  more  convincing 
evidence  that  there  are  times  when  it  does  seem  to  work. 
(1973,  p.  939) 

Such  a  stance  clearly  indicates  the  need  for  simulation 
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research  workers  to  begin  an  investigation  of  the  process 
dimension  of  a  simulation. 

ETHNOGRAPHY:  A  CONCEPTUAL  SCHEME 

In  past  education  research,  preference  appears  to  have 
been  given  to  the  statistical  analysis  of  group  data. 
Researchers  have  tended  to  fit  many  problems  to  the  statistical 
approach.  There  is  evidence,  however,  in  recent  literature  to 
indicate  a  growing  shift  toward  a  broader  conception  of 
appropriate  research  methods.  Mitra  (1974)  has  described  the 
emphasis  on  the  scientific  method  and  in  particular  the  use 
of  statistical  and  psychometric  techniques  as  an  unhealthy 
development  in  educational  research.  The  quantitative, 
statistical  orientation  which  has  come  to  permeate  the  entire 
field  of  educational  research  has  resulted  in  a  study  of 
problems  which  are  statistically  easy  to  handle  but  which  have 
relatively  little  significance  and  value  for  the  discipline 
of  education.  Mitra  has  called  for  a  "systematic  study  of 
significant  problems  rather  than  a  scientific  study  of 
insignificant  problems".  It  is  most  important,  according  to 
Mitra,  to  strike  a  balance  and  produce  research  which  is 
worthwhile,  significant,  relevant  and  understandable  to  those 
willing  and  able  to  bring  about  change  in  education.  Such 
research  can  be  controlled,  reliable  and  valid  without  being 
totally  statistical. 

Of  the  various  nonstatistical  approaches,  the 
methodology  from  anthropology  known  as  ethnography  is  being 
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considered  as  a  promising  alternative  to  research  which 
relies  heavily  on  statistical  analysis.  Ethnography  literally 
means  "writing  about  nations",  derived  from  the  Greek  noun 
"ethno"  which  is  usually  translated  as  "nation  or  tribe  of 
people"  and  the  Greek  verb  "graphy",  translated  "to  write". 
Erickson  (1973)  has  stated  that  any  social  network  forming 
a  corporate  entity  in  which  social  relations  are  regulated  by 
custom  can  serve  as  a  unit  of  ethnographic  analysis.  The 
school  classroom  is  more  frequently  being  considered  as  one 
such  social  unit. 

Shaver  and  Larkins  (1973)  have  described  the  dimensions 
of  the  instructional  setting  for  the  social  studies  as  more 
complex  than  that  for  mathematics  and  the  exact  sciences. 

The  social  studies  have  more  interrelated  cognitive,  affective 
and  interpersonal  behavior  objectives.  Shaver  and  Larkins 
have  considered  ethnography  a  method  of  research  which  may 
provide  more  knowledge  about  the  intricate  interrelations 
among  teachers,  students  and  their  environment  than  the 
traditional  statistical  research  approach. 

The  approach  of  the  ethnographer  is  very  different 
from  that  of  the  statistical  researcher.  The  statistical 
researcher  strives  toward  objectivity,  the  removal  of  his 
influence  from  data  and  findings,  and  as  a  result  separates 
himself  as  much  as  possible  from  students  and  their  social 
context.  He  tends  to  investigate  a  limited  sample  of 
behavior — that  which  can  be  measured,  controlled,  manipulated 
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and  analyzed. 

The  orientation  of  the  ethnographer  is  in  sharp 
contrast.  The  ethnographer  is  immersed  in  direct  and 
continuous  contact  with  the  objects  of  investigation.  His 
approach  is  more  holistic  as  he  attempts  to  examine  behavior 
in  a  broad  context.  The  advantages  of  such  an  approach  have 
been  summed  up  by  Shaver  and  Larkins  as  follows: 

The  ethnographer's  continuous  involvement  in  the 
setting  is  more  conducive  to  looking  at  process  than  is 
the  measurement  of  static  chunks  of  behavior  sliced  off 
prior  to,  during  and  after  treatment.  (1973,  p.  1256) 

In  his  analysis  of  the  contrasts  between  the  ethno¬ 
graphic  and  statistical  research  approaches,  Erickson  (1973) 
has  recognized  ethnography  to  be  an  inquiry  process  rather 
than  a  reporting  process.  The  ethnographer  is  guided  by  a 
point  of  view  and  implicit  questions  which  he  brings  with  him 
to  the  field  rather  than  by  a  set  of  standard  techniques. 

The  ethnographer's  questioning-in- the-field  results  in  a 
description  of  the  regularities  in  social  behavior  as  he 
experiences  and  views  the  whole  social  situation.  The 
classroom  ethnographer  attempts  to  determine  the  inside  view 
of  students,  for  whom  the  same  classroom  events  may  look 
very  different. 

The  ethnographer's  approach  has  been  recognized  as 
more  appropriate  for  the  generation  of  theory  about  instruc¬ 
tion  than  that  of  the  statistical  researcher.  In  a  comparison 
of  the  functions  of  the  two  approaches,  Shaver  and  Larkins 
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The  characteristics  of  the  statistical  tradition — 
the  stress  on  a  carefully  narrowed  scope  of  inquiry, 
predetermined  variables,  and  precise  objective  instru¬ 
ments — make  it  well  suited  for  hypothesis  testing  and 
theory  verification.  On  the  other  hand,  the  ethno¬ 
grapher's  immersion  in  the  object  of  study,  his 
deliberate  attempts  to  be  sensitive  to  many  things,  and 
his  open  search  for  unanticipated  relationships  between 
persons  and  systems  are  well  suited  to  the  identification 
of  important  variables  and  their  antecedents  and 
consequences — the  basis  of  theory  generation. 

(1973,  p.  1257) 

In  their  description  of  the  ethnographic  method, 
Spradley  and  McCurdy  (1972)  pointed  out  that  the  ethnographer 
seeks  consciously  to  be  more  objective  and  to  represent 
accurately  what  people  know  and  believe.  However,  it  must  be 
recognized  that  between  the  events  of  social  behavior  and  the 
investigator's  descriptive  account  is  an  important  variable — 
the  investigator  himself.  In  the  words  of  Spradley  and 
McCurdy : 


In  any  research,  selective  observation  and  selective 
interpretation  always  work  to  transform  the  'actual 
events'  into  'facts'  that  are  used  in  a  descriptive 
account.  Complete  objectivity  may  be  the  characteristic 
of  some  omniscient  observer,  but  not  of  a  human  being. 
(1972,  p.  13) 

Spradley  and  McCurdy  recognized  the  importance  for  an 
investigator  to  realize  the  problems  of  selective  observation 
and  interpretation  and  to  be  conscious  of  the  influences  of 
social  behavior  on  his  study. 

The  school  is  indeed  far  more  complex  than  any  ethno¬ 
grapher's  description  of  it.  According  to  Erickson  (1973) 
the  ethnographer's  descriptions  are  "caricatures  or  systematic 
distortions".  The  ethnographer  deliberately  selects  and 
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reports  details  which  allow  him  to  get  his  point  across. 
Erickson  has  stressed  the  need  for  ethnography  to  provide 
back-up  data  to  illustrate  the  decisions  made  during  the 
research  process,  descriptions  of  the  kinds  and  amounts  of 
data  not  available  and  examples  of  available  data  which  are 
inconsistent  with  the  overall  point  of  view  presented  in  the 
ethnography.  Further  he  has  acknowledged  the  ethnographic 
method  to  be  one  of  "disciplined  subjectivity"  rather  than 
"objectivity" . 

Ethnography  has  been  recently  subject  to  intensive 
methodological  study  and  investigation.  According  to  Wallace 
(1970) ,  ethnography  is  a  description  of  social  interaction 
which  is  the  product  of  cognitive  sharing  and/or  nonsharing. 
Wallace  described  individuals  as  each  having  many  cognitive 
maps  (goals,  plans  and  rules  concerning  appropriateness  of 
behavior)  for  every  social  situation.  The  job  of  the 
ethnographer  today  is  to  infer  the  system  of  rules  which 
people  apply  to  social  situations  rather  than  to  tabulate 
frequencies  and  intercorrelations  of  observed  behaviors.  In 
the  view  of  Wallace,  the  task  of  the  ethnographer  today  is  to 

produce  primarily  not  naturalistic  or  statistical 
descriptions  of  regularities  in  overt  behavior  but 
descriptions  of  the  rules  which  actors  are  presumably 
employing  or  attempting  to  employ  in  the  execution 
and  mutual  organization  of  this  behavior.  (1970,  p.  41  ) 

The  "new  ethnography",  according  to  Sturdevant  (1964), 
aims  at  the  inference  of  rules  and  plans  of  one's  cognitive 
map  structure  from  verbal  and  non-verbal  behaviors.  The 
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process  of  ethnography  requires  the  use  of  categories  based 
upon  informants'  experiences  of  social  situations  rather  than 
the  ethnographer's  own  preconceived  categories.  Cognitive 
anthropologists  have  emphasized  the  need  for  ethnography  to 
be  a  description  of  an  inside  rather  than  an  outside  view 
of  members  of  a  social  situation.  The  ethnographer  today 
must  ask  those  he  is  studying  to  become  his  teachers  and 
instruct  him  in  the  ways  of  life  which  they  find  meaningful. 

Spradley  and  McCurdy  (1972)  have  maintained  that 
ethnographic  research  should  be  conducted  from  the  perspective 
of  cultural  scenes.  They  have  defined  "cultural  scene"  as 
"the  information  shared  by  two  or  more  people  that  defines 
some  aspect  of  their  experience".  A  cultural  scene  is  the 
knowledge  which  actors  employ  in  a  social  situation.  Such  a 
definition  is  based  on  a  cognitive  conception  of  culture 
which  excludes  observable  patterns  of  behavior  and  restricts 
the  culture  concept  to  ideas,  beliefs  and  knowledge. 

In  a  presentation  of  student  ethnographies,  Spradley 
and  McCurdy  have  illustrated  how  an  ethnographer  presents  a 
vivid  description  of  the  rules  and  plans  which  people  use 
daily  in  order  to  achieve  desired  ends.  These  rules,  which 
are  learned  through  symbolic  communication  or  inference  from 
behavior,  determine  both  the  range  and  pattern  of  human 
behavior.  According  to  Spradley  (1972)  these  rules  are 
learned  at  different  levels:  knowing,  understanding,  believing 
and  internalizing.  Individual  variations  in  behavior  are  the 
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result  of  the  levels  at  which  rules  are  learned. 

Among  the  social  situations  which  have  been  described 
by  Spradley  and  McCurdy  are  a  junior  high  school,  Jehovah's 
Witnesses,  a  third-grade  class,  kitchen  culture,  a  second- 
grade  recess  and  fire-fighters.  Variations  of  behavior 
within  any  of  these  social  situations  may  be  attributed  to 
the  rules  which  individuals  have  learned.  Rules  are 
associated  not  only  with  ideal  behavior  (what  individuals 
feel  should  be  done)  but  also  with  presumed  behavior  (what 
individuals  of  a  cultural  group  think  others  do) .  As 
Spradley  has  pointed  out,  both  ideal  and  presumed  behaviors 
may  differ  from  real  behavior  (what  individuals  actually  do) . 
Spradley  has  contended  that  social  interaction  is  largely 
dependent  upon  the  presumed  behavior  of  others  and  therefore 
acknowledged  the  notion  of  presumed  behavior  as  a  helpful 
tool  in  an  analysis  of  cultural  scenes. 

Smith  and  Geoffrey  (1968)  conducted  a  major  classroom 
ethnography  in  an  attempt  to  find  out  how  middle-class 
teachers  cope  with  lower-class  elementary  school  children. 

The  study  identified  numerous  data  sources.  A  non-participant 
observer  (the  investigator)  provided  an  outside  view  of 
classroom  transaction  via  written  descriptions  of  classroom 
events,  interpretive  comments,  audiotaped  summaries  and 
classroom  documents.  A  participant  observer  (the  teacher) 
provided  an  inside  view  of  intended  and  actual  class  trans¬ 
actions.  Students  were  also  recognized  as  potential 
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participant  observers  and  important  data  sources.  Smith  and 
Geoffrey  used  their  data  to  generate  concepts  such  as  pupil 
esteem  and  teacher  awareness  and  to  develop  hypotheses  for 
verification  by  statistical  research. 

The  research  conducted  by  Smith  and  Geoffrey  indicated 
the  need  for  the  ethnographer  to  enter  his  study  with 
questions  in  mind.  Smith  described  these  questions  as 
"foreshadowed  problems"  which  guide  the  ethnographer's 
perceptions  and  provide  a  focus  for  observation.  Such 
questions  are  very  different  from  formal  hypotheses  in 
statistical  research  which  serve  to  structure  the  research 
setting  and  screen  out  unanticipated  data. 

It  would  appear  that  ethnography  and  formative 
evaluation  make  a  successful  marriage.  According  to  Shaver 
and  Larkins  (1973) ,  ethnography  fits  well  into  what  Scriven 
(1967)  refers  to  as  "seat  of  the  pants"  research  which  is 
necessary  in  the  formative  stages  of  curriculum  development. 
Since  formative  evaluators  must  be  attentive  to  unanticipated 
student  and  teacher  reactions  to  aspects  of  the  curriculum 
being  evaluated,  it  would  appear  that  the  ethnographic 
approach  is  most  appropriate  at  this  stage  of  evaluation. 

The  rigorous  designs  and  objectivity  of  the  statistical 
research  approach  might  be  more  appropriate  in  summative 
evaluations  of  the  worth  of  finished  products. 
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SUMMARY 

This  chapter  has  dealt  extensively  with  three  themes 
pertinent  to  the  present  investigation.  In  the  first,  a 
conceptual  scheme  for  formative  evaluation  during  program 
development  has  been  discussed.  In  the  second  theme,  the 
literature  and  research  related  to  simulation  evaluation  has 
been  reviewed.  In  the  third,  a  conceptual  scheme  for 
classroom  ethnography  has  been  presented.  Subsequent  chapters 
will  deal  with  the  manner  in  which  these  conceptual  schemes 
became  operationalized  in  a  formative  evaluation  during  the 
development  of  Transition. 
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CHAPTER  3 


THE  DEVELOPMENT  OF  THE  TRANSITION  SIMULATION 

Conducting  a  formalized  formative  evaluation  of 
Transition  was  the  major  objective  of  this  investigation. 
Nevertheless  the  author  considered  it  important  to  present 
a  preliminary  account  of  the  development  of  the  simulation. 
The  present  chapter  is  an  attempt  to  present  the  rationale, 
framework  and  actual  procedures  for  the  development  of 
Transition.  This  chapter  is  divided  into  the  following 
sections : 

1.  The  Development  Context 

2.  The  Development  Model. 

3.  The  Simulation  Model. 

4.  The  Construction  of  the  Transition  Simulation. 

THE  DEVELOPMENT  CONTEXT 

Transition  was  developed  as  part  of  a  curriculum 
package  of  Project  SURT.  SURT  (The  Study  of  Urban  Rural 
Transition)  is  one  of  fourteen  subprojects  of  Project  Canada 
West,  a  regional  curriculum  development  program  for  the  four 
western  provinces  established  in  April,  1970,  under  the 
Canada  Studies  Foundation.  Project  Canada  West  recognized 
the  primacy  of  the  teacher  as  initiator  and  developer  of  new 
Canadian  Studies  curricula  and  adopted  as  its  theme  the 
'continuing  Canadian  concern'  of  urbanization.  Project  SURT, 
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under  the  broad  umbrella  of  ' urbanization' ,  adopted  the 
specific  theme  of  transition  from  one  community  to  another. 

Project  SURT  focused  on  two  basic  research  questions 
associated  with  urbanization,  namely:  What  does  one  need  to 
know  about  a  community  in  order  to  live  successfully  as  a 
citizen  in  that  community?  How  does  a  person  adapt  to 
successful  living  in  a  different  community?  SURT  recognized 
that  these  questions  are  faced  by  those  who  migrate  from 
urban  to  rural  or  rural  to  urban  Canada. 

Canada  is  composed  of  many  communities  of  varied  sizes 
and  life  styles.  Project  SURT  views  each  of  these  communities 
as  a  social  system  composed  of  many  interacting  subsystems: 
the  family  system,  the  recreation  system,  the  communication 
system,  the  economic  system,  the  religious  system,  the 
education  system,  the  government  system  and  the  services 
system.  The  community  is  not  a  static  entity  but  rather  a 
dynamic  system  of  interacting  subsystems.  Project  SURT  takes 
a  socio-cultural  dynamic  approach  to  Canadian  society.  Since 
its  inception  SURT  has  attempted  to  develop  instructional 
materials  which  will  be  instrumental  in  helping  students 
assess  the  changes  occurring  within  Canadian  communities  and 
the  changes  which  occur  as  people  make  a  transition  from 
rural  to  urban  or  urban  to  rural  centres. 

The  author's  involvement  with  the  SURT  team  began 
officially  in  September,  1973.  As  a  graduate  student  in  the 
Department  of  Secondary  Education  at  the  University  of  Alberta, 
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the  author  received  a  Graduate  Teaching  Assistantship  from 
the  university  which  required  him  to  become  an  active  member 
of  the  SURT  curriculum  development  team.  Shortly  thereafter, 
three  meetings  took  place  to  acquaint  new  members  with  the 
project's  purpose,  past  activities  and  future  plans.  In 
early  October  the  twelve  member  SURT  team  was  divided  into 
two  sub-committees:  one,  consisting  of  four  teachers  and 
five  students,  would  attempt  to  develop  case  studies;  the 
other,  consisting  of  three  new  members,  would  attempt  to 
develop  a  simulation.  The  simulation  development  team 
consisted  of  Jane  Sterk,  a  graduate  of  the  Master  of  Education 
program  at  the  University  of  Alberta  whose  graduate  thesis 
involved  the  design  of  a  simulation  incorporating  the  modes 
of  inquiry  of  Charles  Morris,  and  Bjarne  Myhre  and  the  author, 
both  graduate  students  in  the  Department  of  Secondary 
Education  at  the  time.  It  was  the  decision  of  the  SURT  team 
that  both  the  case  studies  and  the  simulation  would  be  part 
of  one  curriculum  package  designed  for  students  of  secondary 
school  social  studies. 

On  October  3,  1973,  the  simulation  development  team 
with  its  headquarters  at  the  University  of  Alberta  was 
officially  recognized  by  Project  SURT.  In  mid-October  a 
meeting  of  the  entire  team  was  held  in  order  to  establish  a 
theoretical  framework  which  would  ensure  a  goodness  of  fit 
between  the  case  studies  and  the  simulation.  It  was  decided 
at  that  time  that  all  developers  would  use  the  basic  SURT 
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model  of  the  Community  as  a  Social  System.  All  materials 
developed  by  the  SURT  team  would  focus  on  four  subsystems: 
the  education  system,  the  government  system,  the  economic 
system  and  the  recreation  system,  with  the  family  system 
being  the  common  thread.  Both  case  studies  and  simulation 
would  deal  with  Canadian  families  in  their  specific  relation¬ 
ship  to  the  selected  subsystems  and  their  general  relationship 
to  the  remaining  subsystems  of  the  basic  model.  Figure  1  on 
the  following  page  shows  the  components  of  the  SURT  MODEL. 

THE  DEVELOPMENT  MODEL 

The  model  used  by  the  developers  in  the  construction 
of  Transition  was  an  adaptation  of  the  Johnson-Aoki  (1970) 
Curriculum  and  Instructional  Design  Model.  Johnson  (1967) 
distinguished  between  a  curriculum  development  system  and  an 
instructional  system.  He  defined  curriculum  as  "a  structured 
series  of  intended  learning  outcomes",  (p.  6)  These  intended 
learning  outcomes  (I.L.O.'s)  are  selected  from  the  available 
culture,  the  cultural  content,  and  are  organized  and  structured 
according  to  explicit  criteria.  According  to  Johnson, 
curriculum  is  the  end  product  of  the  curriculum  development 
process . 

Curriculum  serves  as  a  guide  to  instructional 
planning.  Curriculum,  the  output  of  the  curriculum  develop¬ 
ment  system,  becomes  the  input  of  the  instructional  planning 
system.  According  to  Aoki  (1970)  instructional  planning  is 
the  preactive  phase  of  the  instructional  system  and  the  phase 
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Figure  1 


The  Social  System  and  Its  Component  Subsystems 


(PROJECT  SURT,  1971) 
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which  links  curriculum  and  instruction.  Instructional  planning 
has  two  major  components:  (1)  instructional  content  and 
(2)  teaching  strategies.  Instructional  content  consists  of 
two  sub-components:  (1)  the  I.L.O.'s  and  (2)  instrumental 
content.  Aoki  (1970)  stated: 

Instrumental  content  is  the  given,  used  to  guide 
students  to  the  ILO.  Instrumental  content  is  the  object, 
event  or  action  with  which  students  transact  affectively , 
cognitively  and  conatively .  A  good  instrumental  content 
is  seductive,  pregnant  with  the  possibilities  for  involved 
intellection  ....  It  is  the  stuff  with  which  students 
transact,  and  in  this  transaction,  the  instructor  is  not 
a  participant,  but  a  mediator;  he  is  a  stage  manager  and 
a  director,  not  an  actor.  (pp.  5-6) 

Throughout  the  entire  development  process,  the  simulation 

development  team  considered  the  simulation  as  incorporating 

instructional  content  and  teaching  strategies. 

Johnson  (1968)  conceived  of  instruction  as  the 
interactive  phase  of  the  instruction  system.  Instruction  is 
essentially  an  interaction  process  in  which  the  "fundamental 
interaction"  is  the  transaction  between  "an  individual  student 
and  selected  segments  of  his  environment".  These  "selected 
segments"  with  which  the  students  transact  form  a  "display". 

The  role  of  the  instructor  is  to  present  "displays"  and 
control  the  classroom  setting.  Figure  2  shows  Aoki's  (1971) 
Phases  of  Teaching  Model.  Aoki  describes  the  student's 
transaction  with  "displays"  as  being  "complicated  by  the 
mediating  presence  of  an  instructor  and  other  students" 

(p.  10) .  As  well  as  transactions  with  selected  segments  of 
the  environment,  the  instrumental  content  displayed,  there  is 
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also  interpersonal  interaction  which  greatly  affects  the 
affective  climate  in  the  classroom  setting. 

In  the  designing  of  Transition  the  developers  viewed 
themselves  as  instructional  planners.  As  experienced  teachers 
they  were  always  cognizant  of  the  process  of  instruction. 

In  the  very  early  developmental  stages  the  simulation  develop¬ 
ment  team  viewed  students  as  future  participants  in  a 
simulation  experience  rather  than  as  learners.  The  developers 
were  aware  of  the  whole  notion  of  learning  by  experience, 
particularly  by  simulation  experience,  being  seriously 
questioned  by  educators.  The  intended  outcomes  formulated 
during  the  development  of  Transition  could  adequately  be 
labelled  "intended  experiential  outcomes"  rather  than 
"intended  learning  outcomes".  These  I.E.O.'s  were  never 
static  but  rather  were  continually  revised  as  the  development 
proceeded.  While  the  development  team  attempted  to  design 
a  product  which  would  be  instrumental  in  achieving  the  general 
objectives  of  Project  SURT,  the  more  specific  intended 
outcomes  of  Transition  were  continually  subject  to  formative 
evaluation. 

Within  the  context  of  the  Aoki  Model,  the  simulation 
when  actually  piloted  was  viewed  as  a  display  within  which 
there  were  many  displays.  Many  segments  of  the  environment 
were  selected  by  the  developers  in  the  construction  of 
Transition.  When  the  simulation  was  first  piloted  the 


experience  for  all  participants,  including  the  simulation 
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leader,  consisted  of  numerous  transactions  with  selected 
displays  as  well  as  numerous  social  interactions.  Figure  3 
is  an  attempt  to  represent  the  manner  in  which  the  Johnson- 
Aoki  Curriculum  and  Instructional  Design  Model  was  adapted  by 
the  simulation  development  team  as  they  were  involved  in  the 
process  of  instructional  planning. 

THE  SIMULATION  MODEL 

The  first  of  many  regular  weekly  meetings  of  the 
development  team  took  place  on  October  24th,  at  which  time  it 
was  recognized  that  the  core  of  any  simulation  is  the  basic 
model.  At  this  initial  developmental  stage  it  was  decided 
to  use  the  method  of  inquiry  postulated  by  Charles  Morris 
(1964)  as  the  basic  model  for  Transition.  According  to 
Morris,  inquiry  is  a  'reflective  process  involving  signs  and 
directed  to  solving  a  problem  (p.  26) .  Morris  identified 
three  types  of  inquiry:  designative  inquiry,  which  deals  with 
problems  of  what  has  happened,  is  happening  or  will  happen; 
appraisive  inquiry,  which  deals  with  problems  of  what  should 
be  or  what  to  accord  preferential  behavior;  and  prescriptive 
inquiry,  which  deals  with  problems  of  what  should  be  done. 

At  any  stage  of  a  problem  solving  situation,  inquiry 
is  never  solely  designative,  appraisive  or  prescriptive. 
Rather  there  is  always  a  designative,  appraisive  or  prescrip¬ 
tive  thrust.  At  any  stage  in  solving  a  problem  there  is 
always  a  predominant  mode  of  inquiry  and  supportive  modes  of 


* 


8. 


- 


' 

. 


■ 

?  29DOTC  11 


Ldoiq  ri-2 x ’.-)  8l6£?b  rJoi.tw  , ^ itf pni.  evisiaaqqft 


n  1  'Liz  '  •' 


: 

q  jv  -;i  q  •  ,  <  b  &  .  Ls  ii  *  ladi  TL&tiJtfl 


■ 


42 


Figure  3 


An  Adaption  of  the  Johnson-Aoki  Model 
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inquiry.  Figure  4  depicts  the  possible  configurations. 


Predominant 
Mode  of 
Inquiry 


Suppor 
Mode  of 
Inquiry 


Figure  4 

Three  Inquiry  Configurations 
(Aoki,  1971;  Sterk,  1971,  p.  33) 

In  order  for  an  inquirer  to  ask  prescriptive  questions,  for 
example,  he  must  have  determined  the  existing  qualities  of  a 
situation  and  must  have  decided  that  a  more  desirable  state 


was  needed. 
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Morris'  method  of  inquiry  was  incorporated  as  the 
basic  model  of  Transition.  In  the  context  of  the  simulation, 
participants  would  engage  in  inquiry  which  had  a  designative 
thrust  when  they  examined  their  present  situations  as  defined 
by  role  cards,  information  booklets  and  other  resource 
materials.  Participants  would  engage  in  inquiry  which  had 
an  appraisive  thrust  when  they  examined  what  they  wanted  to 
have  happen  in  the  simulation  and  made  a  decision  as  to  the 
major  and  specific  goals  they  would  try  to  achieve. 
Participants  would  engage  in  inquiry  which  had  a  prescriptive 
configuration  as  they  inquired  into  the  means  to  achieve 
goals  in  given  situations.  From  the  very  outset  the  develop¬ 
ment  team  perceived  simulation  participants  as  inquirers  in 
an  environment  organized  to  simulate  some  of  the  major 
problems  of  Canadian  families  in  transition  from  rural  to 
urban  or  urban  to  rural  centres. 

As  well  as  a  structured  inquiry  process,  the  develop¬ 
ment  team  recognized  another  crucial  component  in  the 
simulation,  namely  error  control.  The  developers  viewed 
error  control  as  a  stage  of  evaluation  designed  to  analyze 
goals  and  actions  taken  to  achieve  goals  so  as  to  facilitate 
better  decision-making.  In  Transition  participants  would  be 
forced  to  make  decisions  that  would  involve  either  moving 
from  their  present  locations  to  other  communities  or  remaining 
in  their  present  locations  and  adjusting  their  goals.  In  the 
initial  stages  of  development  the  team  recognized  the 
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importance  of  including  opportunities  for  debriefing  at 
selected  stages  of  the  simulation  activities.  During  these 
intermediate  and  final  debriefing  rounds,  participants  would 
be  given  opportunity  to  verbalize  their  experiences  and 
organize  those  experiences  for  subsequent  rounds  of  decision¬ 
making  . 


THE  CONSTRUCTION  OF  THE  TRANSITION  SIMULATION 

Once  decisions  had  been  made  regarding  the  basic 
model  and  component  processes  to  be  incorporated  into 
Transition,  the  development  team  began  the  slow  process  of 
research  and  investigation  of  information  that  had  the 
potential  for  simulation  displays.  Formative  evaluation 
became  an  important  part  of  this  developmental  process  as  the 
developers  explored  and  continually  revised  materials  and 
ideas . 

One  of  the  early  tasks  was  the  identification  of 
family  types  (ranging  in  size,  membership  age,  ethnic  and 
religious  composition) ,  occupations,  housing  facilities  and 
communities  (varying  in  size  and  life  styles)  which  would  be 
representative  of  the  diversified  Canadian  ways  of  life. 

Then  began  the  month  long  process  of  research  into  the 
selected  subsystems.  During  that  time  many  provincial 
government  agencies,  including  Departments  of  Education; 
Culture*  Youth  and  Recreation;  Industry  and  Commerce;  Regional 
Services  and  Building  Development  were  frequented  in  efforts 
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to  procure  as  much  information  as  possible  relative  to  the 
variations  in  economy,  education,  government  and  recreation 
across  Canada.  Responsibility  for  information  gathering  was 
shared  among  team  members.  Regular  weekly  meetings  were 
held  during  which  time  materials  were  presented,  scanned  and 
organized  for  possible  inclusion  in  the  simulation.  These 
early  stages  of  development  reflected  an  open,  flexible 
atmosphere  where  all  suggestions  as  to  how  to  incorporate  the 
vast  amounts  of  information  gathered  into  the  simulation  were 
considered.  Indeed  it  was  such  an  open  atmosphere  within  the 
team  which  facilitated  the  necessary  formative  evaluation 
during  the  developmental  process. 

During  the  third  month  developers  were  preoccupied  with 
visits  to  Statistics  Canada  for  information  on  population  and 
living  conditions.  Letters  were  written  to  the  Department 
of  Urban  Affairs  as  well  as  to  specific  community  government 
centres  requesting  information  on  community  life.  During  this 
stage,  five  communities  were  selected  to  represent  varied 
Canadian  community  life  styles  in  the  simulation.  The 
University  of  Alberta  libraries  were  scanned  for  printed 
materials  on  the  theme  of  urbanization.  The  Curriculum 
Materials  Centre  was  a  locus  of  activity  as  outline  maps  for 
agriculture,  industry,  climate,  transportation,  and  natural 
resources  were  prepared.  First  term  activities  ended  with  a 
December  meeting  of  the  entire  SURT  team  at  which  time  the 
goodness  of  fit  between  the  products  of  the  two  development 


yj olet'W 

. 


. 

.  -  •"■■■  -  '  '•■ 

;  tew  c.ii,‘  a.! aval  d3< fom  trxirid  ©ffcf  ^ni’XiiC 

- 


'  . 

■  ...  •  •'  -'• 

'  - "  •  ?  -.iT*  •  ■-  -  '  ' 


47 


teams  was  the  focal  point  of  discussion. 

It  was  not  until  the  fourth  month  of  the  develop¬ 
mental  process  that  the  specific  intended  experiential 
outcomes  of  the  simulation  were  identified  and  formulated. 
This  was  a  definite  reflection  of  the  unstructured  inquiry 
process  which  had  characterized  the  development  team  as 
members  "muddled  through"  materials.  A  deadline  of  April 
30th  was  set  by  which  time  simulation  materials  had  to  be 
ready  for  piloting.  The  developers  then  undertook  the  task 
of  simplification  of  materials  relative  to  the  four  sub¬ 
systems  in  the  five  selected  communities  so  as  to  be  meaning¬ 
ful  to  students  of  secondary  school  social  studies  and  usable 
within  a  simulation  context.  Considerable  time  was  spent 
planning  the  many  ways  that  materials  could  be  displayed  so 
as  to  involve  simulation  participants  and  encourage  extensive 
transaction. 

In  the  sixth  month  decisions  were  made  as  to  the 
means  of  building  extensive  participant  interaction  into  the 
simulation  as  well  as  the  win-loss  factor  and  resource  units. 
Among  the  issues  discussed  at  this  stage  were  the  pros  and 
cons  of  competition  and  cooperation  in  the  simulation  and  the 
advantages  and  disadvantages  of  fictionalizing  simulation 
materials.  In  the  seventh  and  final  month  of  development 
all  materials  were  translated  into  fictional  terms.  Regional 
and  community  profiles  were  again  subject  to  formative 
evaluation  as  were  the  role  descriptions.  Actual  simulation 
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activities  were  delineated.  In  the  final  three  weeks  the 
developers  shared  the  responsibility  of  printing  and 
reproducing  all  simulation  materials  in  readiness  for  the 
first  piloting. 

The  development  of  Transition  in  its  first  draft 
involved  seven  months  of  teamwork.  Regular  team  meetings 
consumed  165  hours  with  at  least  that  time  again  devoted  to 
the  research  and  investigation  of  materials  by  each  team 
member.  Throughout  the  entire  developmental  process  the 
simulation  development  team  was  engaged  automatically  in 
formative  evaluation  as  team  members  shared  ideas  among 
themselves,  the  entire  SURT  team,  among  consultants  and 
those  not  directly  involved. 

SUMMARY 

This  chapter  attempted  to  indicate  the  context  within 
which  Transition  was  developed,  the  model  used  to  construct 
the  simulation,  the  basic  model  incorporated  into  the 
simulation  and  the  procedure  used  in  the  actual  construction. 
Transition  was  developed  as  part  of  the  curriculum  package 
of  Project  SURT.  It  was  specifically  designed  for  Canadian 
students  of  secondary  school  social  studies.  The  development 
of  the  simulation  came  under  the  category  of  instructional 
planning,  specifically  the  activity  of  designing  instrumental 
content.  The  display  function  of  instructional  materials  was 
considered  especially  important  to  the  development  of 
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Transition. 

The  basic  model  incorporated  into  the  simulation  was 
the  method  of  inquiry  postulated  by  Charles  Morris.  As  well 
as  a  structured  inquiry  process,  the  process  of  error  control 
or  debriefing  was  built  into  several  stages  of  the  simulation 
activities.  From  the  initial  to  the  concluding  stages  of 
the  developmental  process,  the  process  of  formative  evaluation 
was  an  on-going  one.  Product  information  was  continually  fed 
back  into  the  developmental  process  for  the  purpose  of 
improving  the  product. 


. 
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CHAPTER  4 


FORMATIVE  EVALUATION 


The  present  investigation  was  designed  primarily  to 
be  a  formalized  formative  evaluation  of  a  simulation. 
Development  of  any  instructional  content  and  teaching 
strategies  necessitates  that  some  complementary  process  of 
evaluation  occur  during  the  developmental  stages.  Such 
formative  evaluation  is  indeed  crucial  to  the  process  of 
development  and  at  least  equals  the  importance  of  empirical 
evaluations  of  finished  products.  Indeed,  formative  evalua¬ 
tion  is  a  continuing  process  from  the  initial  stages  of  the 
developmental  process.  There  is  also,  however,  an  important 
place  for  a  more  formalized  evaluation*  during  the  stages  of 
development . 

This  chapter  deals  specifically  with  the  manner  in 
which  the  conceptual  scheme  for  formative  evaluation 
presented  in  Chapter  2  became  formally  operationalized 
during  the  development  of  Transition.  This  chapter  is 
divided  into  the  following  sections: 

1.  Formative  Evaluation:  Preactive  Phase 

2.  Formative  Evaluation:  First  Interactive  Phase 

3.  Formative  Evaluation:  Second  Interactive  Phase 

4.  Formative  Evaluation:  First  Postactive  Phase 

5 .  Summary 


*Henceforth  referred  to  as  formative  evaluation, 
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FORMATIVE  EVALUATION:  PREACTIVE  PHASE 

The  goal  of  formative  evaluation  is  not  to  compare 
one  product  or  strategy  with  another  but  rather  to  discover 
deficiencies  and  successes  in  the  intermediate  versions  of 
instructional  content  and  strategies.  The  basic  question  for 
which  the  investigator  had  first  to  find  an  answer  was  how  to 
discover  the  deficiencies  and  successes  of  Transition.  It 
was  decided  at  the  outset  that  the  logical  though  often 
neglected  focus  of  an  investigation  of  Transition  in  its 
formative  stages  of  development  should  be  a  description  and 
assessment  of  the  process  dimension  of  the  simulation.  It 
appeared  most  worthwhile  to  focus  on  the  simulation  partici¬ 
pants  as  they  transacted  with  the  simulation  display  and 
interacted  in  decision-making.  It  was  felt  that  such  an 
investigation  would  provide  the  most  feedback  into  the 
developmental  process  in  order  to  enable  the  development 
team  to  make  necessary  revisions. 

The  investigator  felt  that  there  were  three  basic 
components  of  Transition ,  each  worthy  of  formative  evaluation 
in  its  own  right:  a)  the  simulation  model;  b)  the  sequence 
of  operations,  and  c)  the  scenario  of  materials.  In  recog¬ 
nition  of  the  simulation  model  being  the  core  of  Transition , 
the  investigator  considered  it  a  fundamental  role  of  the 
evaluation  to  provide  cues  to  indicate  whether  or  not  the 
Transition  model  had  validity  and  integrity.  The  Transition 
model  was  recognized  to  include  both  the  decision-making 
model,  the  method  of  inquiry  postulated  by  Morris,  and 
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the  system's  model ,  the  SURT  Community  Systems  Model.  The 
test  for  the  validity  of  the  decision-making  model  involved 
a  major  question.  Do  participant  students  use  the  inquiry 
model  in  any  discernable  way  and  find  it  helpful  in  making 
decisions  without  any  formal  introduction  to  it?  The  test 
for  integrity  of  the  SURT  model  involved  another  major 
question.  Does  the  model  allow  a  flow  of  actual  and 
potential  data  in  the  manner  which  approximates  the  real 
world  system  being  simulated? 

A  second  evaluation  component  was  recognized  to  be 
the  sequence  of  activities  or  the  activities  in  which 
Transition  participants  would  be  engaged.  It  was  recognized 
at  the  outset  that  if  students  at  the  secondary  school  level 
were  unable  to  perform  the  activities  of  Transition  then  the 
task  components  had  to  be  revised  and/or  resequenced.  The 
investigator  considered  another  important  role  of  the  evalu¬ 
ation  study  to  be  the  identification  of  simulation  materials 
which  were  incomprehensible  or  which  did  not  allow  oppor¬ 
tunity  for  in-depth  student  involvement  in  decision-making. 

The  investigator  entered  the  study  fully  convinced 
of  the  need  for  an  examination  of  the  actual  simulation 
process  as  a  means  of  determining  what  students  do  when  they 
are  involved  in  a  simulation  rather  than  what  they  have 
learned.  The  ethnographic  approach  was  chosen  over  statis¬ 
tical,  psychometric  approaches  as  being  more  significant  and 
of  greater  value  in  the  achievement  of  the  goal  of  the 
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formative  evaluation,  namely  the  identification  of  aspects 
of  the  Transition  model,  activities  and  materials  which 
needed  reworking  and  revision.  It  was  felt  that  the  ethno¬ 
graphic  approach  might  provide  more  knowledge  about  the 
intricate  transactions  in  the  classroom  setting  than  a 
traditional  statistical  research  approach.  In  this  sense 
the  study  was  viewed  as  an  exploration  into  the  feasibility 
of  the  ethnographic  research  approach  as  a  means  of  formative 
evaluation . 

Of  immediate  concern  to  the  investigator  during  this 
preactive  stage  of  formative  evaluation  was  the  question  of 
where  and  when  the  first  major  piloting  would  take  place. 

Which  classroom  situation  would  provide  the  most  feedback 
into  the  developmental  process?  The  decision  was  made  to 
involve  a  class  of  the  active  teacher  member  of  the  simul¬ 
ation  development  team.  It  was  felt  that  this  would  be  an 
ideal  situation  for  the  discovery  of  "major  bugs"  in  the 
first  draft  of  Transition .  The  teacher-developer,  with  her 
experience  in  simulations  in  general  and  active  involvement 
in  the  development  of  Transition ,  would  be  able  to  recognize 
immediately  some  of  the  major  deficiencies  and  successes  in  the 
first  draft  and  provide  immediate  and  direct  feedback  to  the 
development  team. 

At  the  same  time  it  was  recognized  that  teachers  not 
involved  in  the  developmental  process  would  be  another 
important  source  of  feedback  information.  They  would  be  less 
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likely  to  develop  tunnel-vision  and  be  in  positions  to 
identify  strengths  and  weaknesses  in  the  simulation  which  a 
developer  might  overlook.  It  was  thus  decided  to  have  three 
teachers  not  involved  in  the  developmental  process  pilot 
Transition  in  their  social  studies  classrooms.  Their  open¬ 
ness  to  unanticipated  problems  and  side  effects  would  be  of 
special  value  in  providing  feedback  to  the  development  team. 

The  class  selected  to  participate  in  the  first  major 
piloting  of  Transition  was  considered  by  the  investigator  as 
a  unit  of  analysis  for  an  ethnographic  study.  The  study 
attempted  to  determine  both  the  inside  view  of  participants 
as  they  transacted  with  the  simulation  and  also  the  outside 
view  of  those  not  directly  involved  in  the  actual  simulation 
experience.  Numerous  data  sources  were  chosen  to  comprise 
the  ethnography  in  order  to  provide  as  much  feedback  as 
possible.  These  sources  of  feedback  included: 

1.  Direct  observations  by  the  participant  observer,  the 
classroom  teacher  and  member  of  the  development  team, 

2.  Direct  observation  by  two  non-particpant  observers, 
the  investigator  and  the  third  member  of  the  development 
team, 

3.  Audiotapes  of  the  verbal  behavior  of  selected 
participant  student  groups  during  selected  segments  of 
their  simulation  experience, 

4.  Videotapes  of  the  verbal  and  visual  behavior  of 
selected  participant  student  groups  during  selected 
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segments  of  their  simulation  experience, 

5.  Recorded  interviews  with  six  participant  student 
groups  immediately  after  completion  of  simulation 
activities, 

6.  A  questionnaire  administered  to  all  participants 
following  their  simulation  experience, 

7.  Classroom  documents  including  student  work  sheets, 
goal  setting  sheets,  and  satisfaction  assessment  sheets. 

The  extensive  nature  of  the  study  called  for  careful 
and  detailed  preparations.  The  development  team  first  drew 
up  a  concise  Time  Line  of  Simulation  Activities  (Appendix  C) . 
This  time  line  proved  crucial  for  the  identification  of 
segments  of  activities  to  be  video  and  audio-taped.  The 
time  line  also  facilitated  the  preparation  of  a  Participant 
and  Nonparticipant  Observation  Sheet  (Appendix  A)  on  which 
the  simulation  leader  and  non-participant  observers  were 
asked  each  day  to  describe  each  simulation  activity  identi¬ 
fied  on  the  time  line  as  they  witnessed  them.  Observations 
were  made  in  terms  of  the  simulation  leader  activities,  the 
participant  student  transactions  and  interactions,  and 
particular  simulation  displays.  Observers  were  also  asked 
to  record  their  judgements  as  to  successes  and  deficiencies 
in  the  simulation  as  soon  as  possible  after  each  day's 
activities . 

The  use  of  video  and  audio  equipment  in  the  study 
also  called  for  careful  planning.  Consideration  had  to  be 
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given  to  the  classroom  setting  and  the  positioning  of 
participant  groups  for  the  piloting.  Decision  was  initially 
made  to  use  the  regular  classroom  setting  so  as  to  reduce 
the  possibilities  of  the  Hawthorne  effect  serving  as  a  source 
of  contamination  of  the  study.  It  was  also  decided  to 
position  participant  groups  so  that  the  administrative  groups 
would  be  located  at  the  classroom  extremities  and  the  resident 
groups  would  be  in  the  classroom  centre.  Figure  5  presents 
the  classroom  arrangement  decided  upon  for  the  simulation 
piloting. 

Such  an  arrangement  appeared  to  offer  the  following 
advantages.  Extensive  interaction  between  resident  and 
administrative  groups  could  be  facilitated  when  administra¬ 
tive  groups  were  positioned  on  the  outside  and  separated 
from  each  other  as  much  as  possible.  Having  resident  groups 
in  the  centre  would  encourage  interaction  among  residents 
although  such  interaction  was  not  specified  as  an  essential 
part  of  the  simulation  activities.  This  arrangement  also 
would  facilitate  video  and  audio-taping  during  selected 
segments  of  the  simulation  experience. 

Permission  having  been  granted  for  the  study  by  the 
Edmonton  Public  School  Board  and  the  co-operating  school, 
the  decision  was  made  to  involve  one  particular  grade  nine 
class  whose  timetable  allotted  three  consecutive  double 
periods  (480  minutes  over  three  days) .  It  was  felt  that 
such  a  distribution  of  class  time  would  be  least  detrimental 
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Classroom  Setting  For  Major  Piloting 
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to  the  continuity  of  the  simulation  experience.  The  teacher- 
developer's  own  experience  with  simulation  pilotings  proved 
of  tremendous  assistance  for  the  investigator  at  this  and 
other  stages  of  decision-making. 

May  15,  16  and  17,  1974,  were  chosen  for  the  major 
class  piloting.  These  dates  ensured  that  the  simulation 
materials  would  be  printed  in  time  and  guaranteed  little 
interference  with  the  class's  regular  social  studies  program. 
Having  the  piloting  in  May  also  avoided  conflict  with  school 
examinations  and  year-end  activities.  Because  the  three 
selected  days  were  so  crucial  to  the  revision  process,  it 
was  decided  by  the  development  team  that  an  initial  testing 
of  video  and  audio  equipment  and  basic  simulation  activities 
was  essential.  The  feedback  from  this  testing  is  reported 
in  the  next  section. 

FORMATIVE  EVALUATION:  FIRST  INTERACTIVE  PHASE 

The  first  student  transaction  with  the  simulation 
occurred  at  Vernon  Barford  Junior  High  School,  Edmonton,  on 
Saturday,  May  11,  1974.  Eleven  volunteer  grade  nine  students 
who  were  not  members  of  the  class  chosen  for  the  major 
piloting  were  involved  in  this  initial  testing  of  simulation 
materials  and  recording  equipment.  The  students,  four  boys 
and  seven  girls,  divided  themselves  into  two  groups.  Five 
volunteered  to  play  the  role  of  administrators  and  each  of 
the  remaining  six  agreed  to  represent  a  resident  family. 

The  majority  of  these  students  had  maintained  above  average 
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achievement  throughout  the  school  year  according  to  the 
teacher's  evaluation.  They  were  all  considered  especially 
co-operative  students  and  involved  in  all  school  activities. 

Most  of  the  basic  activities  of  Transition  were 
carried  out  during  the  initial  testing.  A  brief  fifteen 
minute  introduction  describing  the  history  and  the  purpose  of 
the  simulation  was  given.  Students  drew  role  cards  and  spent 
an  hour  reading  materials  in  order  to  gain  familiarity  with 
roles  and  goals.  Students  were  asked  to  make  notations  of 
any  materials  and  procedures  which  were  difficult  to  compre¬ 
hend.  A  ten  minute  round  of  information  gathering  and 
fifteen  minute  intermediate  debriefing  session  were  conducted. 
Following  a  forty  minute  lunch  break,  all  participants  under¬ 
went  another  round  of  information  gathering  and  made  at  least 
temporary  decisions.  As  time  would  not  permit  any  further 
rounds  of  information  gathering  and  decision-making,  the 
activities  concluded  with  a  twenty  minute  final  debriefing. 

Transition:  Feedback  from  First  Interactive  Phase 

The  following  are  selected  segments  of  the  inter¬ 
action  which  took  place  during  the  initial  testing  and  which 
provided  cues  for  determining  successes  and  deficiencies  in 
the  first  draft. 

First  Segment:  Intermediate  Debriefing 
Simulation  Leader: 

What  I  would  like  to  do  now  is  to  discuss  in  the  next 
few  minutes  some  of  the  things  that  generally  happened 
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during  that  second  round  of  information  gathering.  I 
noticed  some  interesting  things  happening  around  the 
community  of  Mishewah  and  I'd  first  like  to  ask  Sandy 
what  she  observed  as  people  came  to  her  for  information. 

1st  Student  Administrator: 

I  felt  that  in  the  second  round  people  would  be 
quite  detailed  with  their  questions.  In  the  first  round 
all  they  really  wanted  to  know  was  how  much  money  they 
could  get  in  Mishewah,  and  in  the  second  round  they 
wanted  to  know  the  type  of  living  that  was  in  Mishewah 
and  the  kind  of  life  and  the  things  that  people  look 
at  after  they've  got  the  job  problem  mainly  figured  out. 

Simulation  Leader: 

How  did  you  handle  your  information? 

1st  Student  Administrator: 

The  community  is  lacking  in  some  of  its  details  but 
then  every  community  is.  And  so  I  decided  I'd  focus  on 
the  best  parts  of  Mishewah  and  I  think  that  that's  really 
the  people  and  the  atmosphere  and  that  the  cost  of  living 
is  less.  And  people  can  have  a  really  good  time  in 
Mishewah  because  it's  small  and  they  get  to  know  each 
other  better. 

Simulation  Leader: 

That's  a  very  interesting  thing  to  have  happen,  and 
I'd  like  to  know — I  don't  know  if  this  will  make  the 
administrators  feel  uncomfortable — but  I’d  like  now  to 
see  what  your  impression  was  of  the  administrators. 
Sharon's  community,  although  she  probably  didn't  tell 
anybody,  is  decreasing  in  size.  That  means  that  people 
are  deciding  to  move  out  of  Mishewah  for  some  reason  or 
another  and  yet  she  seemed  to  be  able  to  persuade  people 
to  come  there.  On  the  other  hand,  Chinaga,  which  is 
growing  .  .  . 

2nd  Student  Administrator: 

It's  losing  and  nobody's  coming  .  .  . 

Simulation  Leader: 

.  .  .  Robert  wasn't  able  to  persuade  anyone.  And  the 

big  powers  are  of  course  the  cities--although  Wastina 
has  had  her  problems  losing  population — and  they  should 
have  attracted  a  lot  of  people.  Indeed  someone  decided 
to  stay  in  Genessee,  someone  decided  to  stay  in  Zargay. 
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I'd  like  the  resident  groups  to  rate  what  you  saw 
in  the  administrators. 

1st  Student  Resident: 

I  figured  out  that  when  I  went  around  and  talked  to 
the  other  people,  they  weren't  as  persuasive  as  some  of 
the  other  people  were.  It  seemed  like  in  Genessee  they 
had  a  pretty  smooth  talker  and  he  kind  of  talked  you 
into  it.  I  went  around  to  the  other  places  and  the  job 
opportunities  weren't  as  good  so  I  decided  to  move  there. 

2nd  Student  Resident: 

Genessee  was  kind  of  persuasive  because  he  knew  his 
facts  and  everything,  but  it  wasn't  always  on  what 
people  told  you  but  on  what  you  wanted  before  you  went 
there . 

Simulation  Leader: 

You  decided  that  you  would  go  to  Mishewah.  What  did 
you  observe  about  the  administrator? 

2nd  Student  Resident: 

She  wasn't  as  persuasive  as  Rick  there  but  she  knew 
what  she  was  talking  about.  She  didn't  tell  me  every¬ 
thing  . 

Second  Segment:  Final  Debriefing  Session 

Simulation  Leader: 

In  this  simulation  we  have  tried  to  build  in  a  method 
of  inquiry,  a  method  of  decision-making.  And  that  method 
of  inquiry  involves  three  types  of  questions.  The  first 
question  you  ask  yourself  is  "What  is?"  or  What  do  I 
have  now?  What  do  I  think  I'll  have  next  year?"  The 
next  question  you  ask  is  "What  do  I  want?"  In  order  to 
set  your  goals  you  decide  what  you  would  like.  And 
finally  you  ask  the  question  "What  should  I  do  to  get 
what  I  want?" 

That's  not  an  adequate  explanation  of  the  inquiry 
process,  but  I  was  wondering  if  anyone  found  himself 
thinking  in  terms  of  those  three  processes.  And  particu¬ 
larly  the  administrators,  I'd  like  you  to  volunteer  to 
say  whether  you  thought  people  were  asking  those  three 
questions . 

3rd  Student  Resident: 

I  was  looking  for  a  job  opportunity  as  a  baker 
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because  the  one  that  I  have  now  didn't  pay  enough  to 
support  my  family. 

Simulation  Leader: 

Sounds  good ,  now  what  did  you  do? 

3rd  Student  Resident: 

Well,  I  went  around  to  the  different  administrators 
asking  them  about  job  opportunities  and  income  rates 
with  jobs.  They  told  me  about  different  jobs  they  had 
and  how  much  they  paid,  and  I  told  them  I  would  come 
back  later. 

Simulation  Leader: 

All  right.  Did  you  make  a  decision? 

3rd  Student  Resident: 

Well,  I  went  to  two  of  the  administrators  and  one  of 
them  was  in  the  place  I  am  living  already.  And  the 
third  one  I  went  to  was  Zargay  and  they  had  what  I 
wanted  ....  So  I  decided  I  would  move  to  Zargay  to 
get  the  better  job. 

Simulation  Leader: 

I'd  like  one  of  the  administrators  to  describe  what 
he  observed. 

3rd  Student  Administrator: 

Some  people  were  really  organized  in  their  questions. 
Sandy  came  and  she  had  a  list  of  questions  about  what 
she  wanted  to  know  about  each  area,  and  she  would  sit 
down  with  her  notes  and  make  a  decision  quickly.  She 
knew  what  she  was  after.  Most  people  would  come  with 
one  question  and  get  an  answer  and  then  they'd  go  and 
ask  other  people.  Then  they'd  come  back  and  ask  another 
question  when  they  could  have  had  all  their  questions 
written  out. 

Simulation  Leader: 

O.K. ,  that  could  have  been  the  result  of  my  direc¬ 
tions.  But  when  they  would  come,  did  you  find  yourself 
asking  them  questions? 

3rd  Student  Administrator: 

Well,  a  lot  of  them  didn't  know  what  they  were 

looking  for,  and  you  had  to  ask  them  what  their  old  job  was, 
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what  they  were  trying  to  get,  what  facilities  they  were 
looking  for.  A  lot  of  people,  even  when  you  asked  them, 
had  to  go  back  to  their  desk  and  check  it  out  and  come 
back  later. 

Simulation  Leader: 

What  about  the  'should  do'  questions?  Did  you  help 
them  with  that? 

3rd  Student  Administrator: 

Yeah,  well  a  lot  of  times  there  was  a  job  where  they 
couldn't  quite  meet  the  qualifications  but  there  were 
educational  facilities.  So  I  advised  them  that  they 
should  get  further  education  and  then  meet  the  qualifica¬ 
tions  for  the  jobs. 


These  selected  segments  contained  many  cues  for  deter¬ 
mining  some  strong  and  weak  points  in  the  simulation  model, 
sequence  activities  and  scenario  of  materials.  The  following 
conclusions  were  drawn  after  the  initial  student  transaction 
with  Transition. 

1.  The  first  selected  segment,  the  intermediate 
debriefing,  strongly  suggested  that  the  simulation 
materials  had  the  potential  for  in-depth  student  involve¬ 
ment.  The  1st  Student  Administrator  appeared  to  under¬ 
stand  her  community  information  and  made  the  decision  to 
focus  on  the  good  points  of  her  community  in  order  to 
attract  residents,  in  particular  the  friendly  community 
feeling.  Other  administrator  groups  went  about  reaching 
the  same  goal  in  other  ways.  It  appeared  that  the 
administrator  materials  offered  a  variety  of  alternatives 
for  handling  information  and  making  decisions.  The 
alternatives  appeared  realistic  and  manageable  to  those 
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administrators  who  participated  in  the  initial  testing. 

2.  The  reports  of  the  1st  and  2nd  Student  Residents 
indicated  that  the  simulation  allowed  for  extensive  and 
involved  role  playing.  Administrator  Materials 
immediately  enticed  some  students  who  became  actively 
involved  in  persuasive  techniques. 

3.  The  2nd  Student  Resident  report  during  the  final 
debriefing  really  brought  home  the  importance  of  goal 
setting.  She  saw  her  goal  as  being  as  influential  as 
the  smooth  talk  of  administrators  in  her  making  a  final 
decision.  This  gave  some  indication  that  the  methods  of 
inquiry  incorporated  into  the  simulation  model  were 
logical  and  workable. 

4.  Although  only  two  rounds  of  information  gathering 
were  conducted,  the  participant  students  evidenced  no 
serious  problems  in  performing  the  sequenced  activities 
of  the  simulation.  The  initial  testing  did  indicate 
that  some  of  the  major  activities  would  take  longer  than 
originally  anticipated  during  the  major  class  piloting. 
Since  time  did  not  permit  student  involvement  in  compuation 
of  scores,  no  cues  were  provided  by  the  initial  testing 
to  determine  if  the  mathematical  formulas  were  complicated 
and  time  consuming. 

5.  The  second  selected  segment  gave  strong  indications 
that  participants  were  indeed  using  the  inquiry  process 
of  Transition 1 s  decision-making  model  in  a  discernable 
way.  It  was  most  interesting  to  observe  the 
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administrators  using  a  structured  inquiry  process  rather 
than  an  unstructured  muddling  process  even  though  the 
simulation  did  not  structure  their  means  of  handling  the 
information  presented.  Developers  interpreted  this  as  a 
sign  that  the  simulation  model  was  valid.  Both  resident 
and  administrator  groups  found  the  model  a  logical  one 
to  use  when  faced  with  a  problem.  Reports  of  both  the 
3rd  Student  Resident  and  the  3rd  Administrator  revealed 
that  their  inquiry  process  had  moved  from  a  designative, 
to  an  appraisive  and  prescriptive  configuration  as  they 
proceeded  to  solve  their  very  different  problems  of 
attracting  more  people  on  the  one  hand  and  finding  a 
better  paying  job  on  the  other. 

6.  What  was  evident  during  the  initial  testing  was 
extensive  enjoyment  and  satisfaction  among  the  partipants 
as  they  set  personal  goals  and  tried  to  achieve  them. 
These  observations  could  have  been  attributable  to  the 
particular  student  personalities,  the  novelty  of  their 
simulation  experience,  and  the  simulation  materials 
themselves . 

7.  One  of  the  major  problems  identified  as  a  result  of 
this  initial  testing  was  the  inadequacy  of  the  regular 
classroom.  Even  with  only  eleven  participating  students 
the  classroom  was  overcrowded  due  to  the  space  taken  up 
by  the  recording  equipment.  This  rendered  the  use  of 
the  regular  classroom  impractical  if  not  impossible  for 
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the  major  piloting  the  following  week  with  a  class  of 
thirty  students.  Arrangements  were  made  immediately 
after  the  initial  testing  to  use  the  school  library  for 
the  major  class  piloting. 

FORMATIVE  EVALUATION:  SECOND  INTERACTIVE  PHASE 

The  second  group  transaction  with  the  simulation  took 
place  at  Vernon  Barford  Junior  High  School  on  Wednesday,  May 
15th;  Thursday,  the  16th;  Friday,  the  17th,  and  the  following 
Wednesday,  May  22,  1974.  Twenty-eight  students  of  a  grade 
nine  class  were  involved  in  this  major  piloting.  The 
students,  fourteen  boys  and  fourteen  girls,  were  assigned  to 
groups  of  two  by  the  simulation  leader.  Five  of  the  pairs 
were  appointed  to  the  administrative  roles  and  the  remaining 
nine  pairs  selected  role  cards  which  assigned  them  to 
resident  family  roles.  The  majority  of  these  students  had 
maintained  average  achievement  throughout  the  school  year 
and  were  considered  by  the  teacher-leader  as  "the  best  grade 
nine  class  in  the  school"  because  of  their  co-operativeness 
and  overall  enthusiasm. 

All  intended  simulation  activities  were  carried  out 
because  the  major  piloting  was  extended  into  a  fourth 
day.  On  the  first  day  an  extensive  introduction  was  given 
by  the  simulation  leader  in  her  description  of  the  develop¬ 
ment  and  purpose  of  Transition .  The  role  of  the  student  in 


formative  evaluation  was  emphasized.  The  meaning  of 


•  .  ■'  ■  ■  ■  '  ; 


rr.  to  I  lori:  2  >t  r  H  : 

P.3.X&C;  Sii3  ©Vi/5 

£  f\  b$&TL3&!  -  f>  Oi  •  c  b&i  Oi  q .  93&V 


,  »  1  •  ■  ■  ^  r.iV 


■ 


67 


simulation  was  explained  via  use  of  the  overhead  and  trans¬ 
parencies  which  compared  simulations  with  athletic  games. 

The  fictionalized  communities  and  regions  were  described  with 
reference  to  a  wall  map.  The  types  of  groups,  roles  and 
general  requirements  of  participants  were  explained.  Fifty 
minutes  were  devoted  to  this  introduction  which  was  mainly 
leader  dominated.  Opportunity  was  provided  for  student 
questioning  and  interaction. 

Participant  manuals  were  distributed  and  students 
spent  twenty  minutes  reading  them.  Specific  activities  were 
clarified  by  the  simulation  leader  with  reference  to  the 
manuals.  Students  then  drew  role  cards  and  assumed  class 
positions.  The  remaining  hour  was  allotted  to  reading  of 
information  packs.  During  this  time  resident  groups  also 
completed  work  sheets  and  set  goals.  Administrative  groups 
set  goals  and  prepared  for  their  class  presentations  to  sell 
their  respective  communities. 

On  the  second  day,  all  five  administrative  presenta¬ 
tions  were  conducted.  The  activities  of  the  preceding  day 
were  reviewed  by  the  simulation  leader.  Two  ten  minute  rounds 
of  information  gathering  were  carried  out,  each  followed 
by  intermediate  debriefing  sessions.  Ten  minutes  were 
devoted  to  the  clarification  and  re-evaluation  of  goals. 

The  third  day's  activities  focused  on  two  additional 
rounds  of  information  gathering  and  debriefing.  The  point 
system  was  clarified  by  the  leader  and  one  hour  was  occupied 
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by  students  as  they  wrestled  with  the  simulation's  mathe¬ 
matical  computations.  Activities  for  this  day  ended  with  all 
resident  groups  having  made  final  decisions. 

During  the  extended  fourth  session,  all  participant 
groups  made  class  presentations  in  an  attempt  to  describe 
what  happened  to  them  during  their  simulation  experience  and 
to  communicate  their  credit  and  satisfaction  point  assessment 
to  the  class.  As  each  group  made  a  presentation,  three 
students  were  designated  to  evaluate  the  group  in  terms  of 
the  realism  of  their  assessment.  Group  evaluations  were 
followed  by  a  general  class  discussion  of  what  happened 
throughout  the  entire  four  day  experience  with  particular 
reference  to  the  simulation  model,  activities  and  materials. 
Questionnaires  were  distributed  and  returned  to  the  leader 
the  following  day.  Interviews  with  twelve  participants  were 
conducted  immediately  following  the  conclusion  of  simulation 
activities.  In  all,  seven  hours  were  required  for  the 
completion  of  all  activities.  The  entire  simulation 
experience  took  place  in  a  carpeted  open  area  of  the  school 
library. 

FORMATIVE  EVALUATION:  FIRST  POSTACTIVE  PHASE 

The  first  postactive  phase  of  formative  evaluation 
began  with  an  analysis  of  ethnographic  data  obtained  during 
the  major  class  piloting.  In  the  following  section  is 
presented  a  summary  and  analysis  of  data  gathered  from  three 
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sources:  participant  interviews,  questionnaire  and 

participant  worksheets. 

Interviews 

Immediately  following  the  final  simulation  activity 
of  the  major  pilot  study,  interviews  were  conducted  with  six 
participant  groups.  Three  resident  groups  were  interviewed 
by  a  developer-observer.  Because  of  technical  difficulties 
with  recording  equipment,  these  interviews  were  confined  to 
a  twenty  minute  period.  Three  administrative  groups  were 
interviewed  by  the  investigator  over  a  forty-five  minute 
period.  Both  interviewers  had  had  previous  direct  contact 
with  informants  during  the  seven  hours  of  observation  of 
their  simulation  experience. 

Interviews  were  conducted  from  an  ethnographic 
perspective.  Each  participant  group  was  viewed  as  a  cultural 
scene  by  virtue  of  the  two  members'  common  experience  with  a 
specific  simulation  role.  Both  members  of  the  group  were 
viewed  as  sharing  information  which  defined  their  common 
experience.  The  interviews  were  an  attempt  to  determine  the 
extent  of  cognitive  sharing  of  goals,  plans  and  rules  during 
the  simulation  experience  and  to  explore  the  reactions  of 
participant  groups  to  that  experience.  The  interviews 
represented  a  means  for  an  examination  of  simulation 
activities  from  an  inside  student  perspective  as  distinct 
from  the  outside  view  of  participant  and  non-participant 


observers . 
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Inquiries  conducted  with  participant  groups  were 
unstructured.  Interviewers  entered  into  dialogue  with 
participants  with  foreshadowed  problems  in  mind.  These 
included  the  following: 

1.  The  extent  of  co-operation  within  and  among  groups. 

2.  The  extent  to  which  materials  and  activities  were 
difficult  and  confusing. 

3.  Participant  attitudes  to  the  inquiry  process  of  the 
simulation  model. 

4.  The  degree  of  satisfaction  and  enjoyment  of 
participants . 

5.  Participant  perceptions  of  the  overall  experience. 
These  problems  served  merely  as  a  guide.  The  actual 
categories  for  discussion  were  based  in  part  on  the  informants* 
experiences  rather  than  totally  on  preconceptions  of  the 
interviewers.  Cues  for  further  discussion  were  picked  up 
from  informants  themselves  as  the  interviews  progressed. 

All  interviews  were  tape  recorded  and  then  tran¬ 
scribed.  Transcriptions  were  analyzed  in  order  to  determine 
participant  perceptions  of  and  attitudes  to  their  simulation 
experience.  The  following  are  selected  segments  which  were 
especially  significant  in  providing  the  investigator  with 
insight  into  the  manner  in  which  grade  nine  students  engage 
in  role  playing  and  decision-making  and  in  providing  the 
development  team  with  cues  for  further  simulation 


revisions . 
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Segments  of  Interview  with  First  Administrative  Group 

Interviewer : 

Did  you  at  any  point  find  the  simulation  difficult 
or  confusing? 

Informant  1: 

Well,  at  the  first  when  we  read  the  information,  I 
didn't  really  understand  the  whole  thing  but  later  I 
understood  it.  At  first  I  thought  it  was  going  to  be 
boring  but  later  I  found  it  to  be  fun. 

Interviewer : 

Do  you  think  you  might  be  involved  in  real  life  with 
some  of  the  kinds  of  decisions  you  made  in  the  simulation? 

Informant  2: 

I  think  you  have  to  think  of  moving.  Getting  a 
house,  better  wages,  a  better  district--you  have  to  look 
at  all  the  possibilities  in  moving.  I  think  when  people 
consider  moving  the  first  thing  they  consider  is  the  job 
and  salary.  But  almost  everything  is  important. 

Interviewer : 

Do  you  think  the  model  for  the  simulation  presents  a 
good  method  for  problem  solving? 

Informant  1: 

I  think  so  because  this  method  helps  you  consider 
all  the  things,  all  the  alternatives.  If  you  consider 
all  the  alternatives  then  you  could  probably  come  to 
better  conclusions.  I  think  that  Mrs.  Sterk  should  have 
stressed  the  problem  solving  method  at  the  beginning 
moreso,  and  then  people  would  have  been  realistic  in 
their  goals.  Some  weren't  realistic  in  their  goals. 

At  first  when  we  were  writing  our  goals  I  didn't  really 
know  and  understand  what  we  were  supposed  to  write  down. 

Interviewer : 

Do  we  always  make  decisions  that  way?  Do  we  always 
look  at  all  the  alternatives? 

Informant  2 : 


No.  A  lot  of  times  we  never  do.  A  lot  of  times 
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there  are  things  which  I  don't  think  are  important  but 
which  really  are.  Basically  if  you  see  one  thing  you 
like  then  you  don't  go  searching  for  other  possibilities; 
you  are  satisfied  with  that. 

Interviewer : 

When  we're  making  decisions,  are  we  influenced  just 
by  facts  or  by  other  things  as  well?  Are  we  influenced 
just  by  information  or  also  by  the  people  who  convey  the 
information? 

Informant  2 : 

I  think  we're  basically  influenced  by  people.  You 
know,  if  they  sound  really  good,  sometimes  they  suck  you 
in. 


Interviewer : 

Can  you  think  of  examples  in  your  own  life? 

Informant  2 : 

Yes,  my  decision  to  baby-sit.  I  really  don't  like 
baby-sitting  but  when  they  say  in  a  nice  voice  "I  really 
need  you"  it  is  sort  of  difficult  to  say  "no". 

Interviewer: 

You  know  the  trend  in  Canada  is  for  people  to  move 
to  the  larger  cities  and  not  communities  of  four  thousand 
people  like  yours.  I  really  wonder  if  it  was  the 
material  or  if  it  was  your  effort  that  you  put  into  it. 

Informant  1: 

Well,  we  tried  to  satisfy  everyone.  People  didn't 
really  look  at  what  the  present  population  was.  They 
kind  of  were  just  looking  for  their  needs.  If  we  did  it 
again  people  would  watch  more  carefully  what  they  were 
looking  into,  like  inflation  rates,  etc. 

Informant  2 : 

I  think  it  was  really  luck.  I  don't  think  we  were 
too  hot  in  selling  the  stuff  but  we  had  a  pretty  good 
community.  We  could  have  probably  done  better  in 
selling  it. 

Interviewer : 

I  was  interested  in  your  handling  of  the  situation 
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the  first  day.  One  group  stayed  with  you  the  whole  time, 
the  whole  ten  minutes  and  yet  you  didn't  seem  worried. 

Why  not? 

Informant  1: 

We  weren't  really  concerned  about  getting  numbers. 

We  wanted  to  make  sure  that  the  people  who  came  to  our 
community  were  those  who  knew  all  about  it  and  really 
wanted  to  come  ....  If  you  spend  more  time  with  people 
then  you  might  convince  them  whereas  if  you  spend  a 
little  time  with  everybody,  then  they  might  not  be  too 
impressed.  We  weren't  really  concerned  about  getting  a 
lot  of  people.  We  let  people  stay  with  us  until  they 
had  all  the  information  they  wanted. 

Interviewer : 

Did  you  feel  at  any  time  over  the  four  days  that  you 
were  competing  with  the  other  administrators? 

Informant  2: 

In  a  way  you  did,  because  when  people  came  they 
would  say  "We  can  get  this  much  in  this  place.  What  can 
you  offer?"  You  had  to  try  to  beat  that.  But  one  group 
decided  to  move  to  us  even  though  their  salary  was  lower . 
We  had  other  advantages. 

Segments  of  Interview  with  First  Resident  Group 

Interviewer: 

What  do  you  think  about  the  game?  What  needs  to  be 
changed? 

Informant  1: 

I'd  say  the  introduction  was  off,  like  you  would  get 
lost  in  the  introduction  and  wouldn't  know  exactly  what 
was  going  on. 

Interviewer : 

Did  you  find  anything  difficult  about  this  game? 

Informant  2 : 

The  jobs.  There  just  wasn't  one  that  suited  us 
perfectly. 
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Interviewer : 

Do  you  think  this  is  the  way  life  is? 

Informant  2: 

Yes,  I  guess  it's  hard  to  find  the  perfect  job. 

Interviewer : 

Would  you  do  anything  differently  if  you  were  to 
play  the  same  role  again? 

Informant  1: 

I  think  we  would  do  some  things  differently.  We 
would  find  out  much  more  about  the  city.  Like  we  were 
just  looking  for  jobs  but  we  weren't  really  thinking  too 
much  about  social  things  and  cultural  stuff. 

Interviewer : 

What  do  you  think  the  purpose  of  this  game  was? 

Informant  2: 

That's  what  I  didn't  get,  the  purpose  of  it  all.  I 
didn't  see  why  we  had  to  do  it. 

Interviewer : 

Do  you  think  it  helped  to  teach  you  how  decisions 
were  made? 

Informant  2 : 

I  just  answered  the  questions  and  I  don't  think  it 
helped  me.  I  enjoyed  the  game  and  found  it  interesting. 

Interviewer : 

What  was  the  most  difficult  part  of  the  game  for  you? 

Informant  1: 

The  rules  were  a  bit  confusing  at  the  beginning. 

When  the  teacher  explained  the  rules  in  the  first  round 
no  one  really  knew  what  they  were  supposed  to  do  and 
what  questions  to  ask.  Maybe  the  teacher  should  give 
some  examples  when  going  over  the  rules.  Rules  should 
be  more  defined  and  better  explained. 
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Interviewer : 

Do  you  think  this  game  will  help  you  in  the  future? 

Informant  1: 

It  shows  you  that  you  should  look  for  everything  in 
a  community  and  not  just  one  specific  thing  because  then 
they  could  be  lacking  in  another  one. 

Interviewer : 

What  do  you  think  the  main  purpose  of  the  game  was? 

Informant  1: 

I  don't  really  know  what  it  was. 

Segments  of  Interview  with  Second  Administrative  Group 

Interviewer : 

Do  you  think  the  game  is  realistic? 

Informant  1: 

There  was  a  lot  of  chance  involved.  In  real  life 
there  is  more  control  over  certain  situations.  We 
couldn't  make  any  deals  like  some  businesses  do  in  real 
life.  We  would  like  to  have  been  able  to  do  more  with 
the  materials.  In  Mishewah  the  jobs  were  all  low  salary. 
What  we  had  couldn't  really  help  any  one  the  way  things 
were.  I  don't  think  that  Mishewah  fitted  into  anyone's 
real  plans. 

Interviewer : 

What  were  your  goals? 

Informant  2: 

To  get  people  to  come  to  our  community  and  give  them 
nice  jobs. 

Interviewer : 

Any  specific  jobs? 

Informant  2 : 

To  see  as  many  people  as  possible. 
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Interviewer : 

Did  you  change  these  goals  in  any  way? 
Informant  1: 


No.  We  stuck  to  them. 


Interviewer : 

How  did  you  go  about  trying  to  achieve  your  goals? 

Informant  1: 

Well  first  the  speech,  and  to  try  and  make  things 
sound  as  good  as  possible.  We  deliberately  selected 
bits  of  information.  If  we  had  told  them  the  truth  that 
would  have  killed  it  right  there. 

Interviewer : 

What  about  the  method  of  decision-making?  Is  it  a 
good  approach  to  solving  problems? 

Informant  1: 

Well  I  would  have  to  say  that  it  is  a  logical 
approach  to  solving  problems  but  ....  It  is  a  basic 
way  to  start  looking  at  things.  Let's  say  that  the  real 
estate  agent  wanted  to  sell  something,  then  he  would 
look  at  what  was  and  then  what  he  wanted. 

Interviewer : 

Were  you  satisfied  with  what  you  had  done  even  though 
you  didn't  achieve  your  goals? 

Informant  1: 

Yes.  I  think  we  did  just  about  what  anyone  could 
really  do. 

Interviewer : 

Did  you  feel  competition? 

Informant  1: 

Yes,  right  off  the  batl  Constantly  trying  to  shuttle 
people  back  and  forth.  Sometimes  I  had  to  remind  myself 
that  it  was  just  a  game,  that  it  wasn't  all  that  serious. 
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Interviewer : 

Would  this  help  you  in  the  future  in  any  way  to  make 
decisions? 

Informant  1: 

This  would  make  me  more  wary  of  administrators.  Now 
I  know  how  they  operate. 

Interviewer : 

When  we  make  decisions  are  we  just  influenced  by 
factual  information? 

Informant  1: 

I  think  people  are  sort  of  trained  today  to  forget 
about  advertising  and  look  basically  at  what  there  is — 
facts. 

Interviewer: 

Do  people  play  a  part  in  your  everyday  decisions? 

Informant  1: 

Someone  has  to  be  there  to  tell  you  about  it.  Slight 
endorsements  do  influence  you  a  Little  bit.  Some  people 
are  gullible. 

Interviewer : 

Were  you  bored  at  any  time  over  the  four  days? 

Informant  1 : 

I  was  always  engaged  into  it  and  working  at  it. 
Materials  were  easy  to  work  with. 


Questionnaire 

A  total  of  thirty-nine  copies  of  a  questionnaire 
(Appendix  B)  were  distributed  to  all  participants  of  both 
the  initial  testing  and  major  class  piloting.  Twenty-three 
of  these  were  returned  to  the  simulation  leader  and  forwarded 
to  the  investigator.  Results  of  the  questionnaire  were 
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tabulated  and  analyzed.  The  following  student  feedback  was 

obtained : 

1.  Twelve  students  felt  that  the  simulation  should  have 
been  longer  while  nine  were  satisfied  with  its  original 
length  and  two  felt  that  the  simulation  was  too  long. 

2.  The  most  common  suggestions  for  additions  to  the 
simulation  were: 

a.  a  better  explanation  of  what  the  game  is  all 
about  (7) , 

b.  the  use  of  examples  during  the  introduction  (7) , 

c.  longer  interaction  rounds  (7) , 

d.  a  more  realistic  balance  of  good  and  bad  in  each 
community  (2) , 

e.  a  package  of  materials  for  each  administrator  (2), 

f.  more  power  for  the  administrators,  such  as  paying 
moving  expenses  or  giving  a  month's  free  rent  (5). 

3.  The  great  majority  of  participants  felt  that  nothing 
should  be  deleted  from  the  game. 

4.  When  asked  via  a  checklist  to  identify  those  things  which 
were  difficult  and  easy  to  understand,  students  reported 


the 

following : 

Difficult 

Easy 

a. 

the  introduction 

12 

9 

b. 

the  background  information 

4 

12 

c. 

the  description  of  the  problem  of 
transition  in  the  world  today 

8 

9 

d. 

the  description  of  the  community 
systems 

3 

14 

e. 

the  rules 

4 

16 

f . 

the  information  on  my  community 

1 

19 
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Difficult  Easy 


g- 

my  role  description 

3 

17 

h. 

the  worksheet 

4 

1 

i . 

the  activity  of  setting  goals 

6 

13 

j- 

the  information  which  resident 
groups  wanted  when  they  came  to 
me  for  information 

1 

7 

k. 

the  information  which  administrative 
groups  had  when  I  went  to  them  for 
information 

4 

10 

1. 

making  a  final  decision 

2 

14 

m. 

the  formulas 

16 

2 

n. 

the  credit  points 

12 

7 

o. 

the  satisfaction  points 

7 

8 

P- 

the  evaluation  of  my  group  by  others 

6 

8 

The  obvious  problem  areas  for  participants  were 

the 

introduction  and  explanation  of  Transition ,  the  mathe¬ 
matical  computations  and  point  system. 

5.  Nineteen  participants  found  their  role  easy  because  the 
information  in  the  simulation  was  well  organized,  precise 
and  understandable  and  because  they  were  satisfied  with 
the  means  they  used  to  achieve  goals.  Three  participants 
who  found  their  role  to  be  difficult  cited  frustration 
and  failure  to  attract  people  and  lack  of  understanding 
of  their  role  as  reasons. 

6.  Fifteen  participants  felt  they  had  played  their  role  well 
because  they  had  carried  out  the  role  obligations  and  had 
reached  their  goals.  Eight  participants  felt  they  had 
played  their  role  poorly  and  cited  the  following  reasons: 

a.  I  didn't  know  what  was  happening. 

b.  We  didn't  gain  much. 

c.  I  made  a  poor  speech  and  loused  up  the  description 
of  my  community. 

d.  I  could  have  done  better  with  more  time. 

e.  I  wasn't  satisfied. 
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7.  Nineteen  participants  felt  that  their  role  was  important 
to  the  success  of  their  group  and  cited  the  following 
reasons : 

a.  Co-operation  and  gameship  are  very  important  for 
success . 

b.  Resident  groups  relied  on  me. 

c.  People  needed  someone  to  convince  them. 

d.  I  informed  as  many  as  my  partner. 

e.  I  had  some  difficult  decisions  to  make. 

f.  My  goals  were  clear  and  well-defined. 

One  participant  noted  that  his  role  was  unimportant 
because  he  didn't  know  what  he  was  doing  at  first. 

8.  Eighteen  students  felt  that  their  role  required  more 
information  so  that  they  could  better  understand  the 
people  they  represented  and  could  better  communicate 
their  role  to  others.  Five  participants  found  the  role 
information  sufficient. 

9.  Nineteen  students  felt  that  their  group  had  an  important 
role  to  play  in  the  game  and  noted  the  following  as 
major  reasons: 

a.  We  had  to  make  real  life  decisions  in  this  game. 

b.  There  was  room  in  the  game  for  our  opinions. 

c.  This  game  shows  what  can  happen  in  today's  life — 
decision-making  and  problems. 

d.  We  helped  people  to  make  decisions. 

e.  If  we  thought  it  was  unimportant  we  would  not  have 
had  a  success. 

f.  Co-operation  is  necessary  for  achievement. 

g.  We  helped  persuade  and  give  information. 

h.  It  was  possible  for  our  community  to  be  beneficial 
to  people. 

The  one  participant  who  reported  his  group  role  as 
being  unimportant  did  not  cite  any  reasons. 


10.  Nineteen  participants  found  the  simulation  to  be  just 
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right  for  their  grade  level  and  noted  the  following 
reasons : 

a.  We  should  start  realizing  the  problems  of  moving  in 
grade  nine. 

b.  The  game  shows  us  some  problems  of  decision-making 
in  today's  world. 

c.  The  game  requires  some  skill  and  a  lot  of  thinking 
but  is  not  overly  hard. 

d.  The  game  really  makes  you  think  but  is  still  a  lot 
of  fun. 

e.  The  game  is  fun,  interesting  and  taught  me  a  little 
more  about  world  problems. 

f.  The  game  provides  us  with  a  challenge. 

The  two  participants  who  found  the  game  too  difficult 
found  the  instructions  too  complicated  for  their  level. 

11.  Thirteen  participants  reported  being  happy  with  their 
final  decision  and  equated  happiness  generally  with 
achievement  of  goals  and  helping  people  improve  their 
situations.  Four  stated  that  they  were  unhappy  with 
their  final  decision  because  they  were  rushed  into 
making  the  decision. 

12.  Twenty  participants  said  they  would  play  the  game  better 
a  second  time  for  the  following  reasons: 

a.  I  would  know  what  to  do  and  would  better  understand 
rules. 

b.  I  would  get  right  into  the  play  faster. 

c.  I  would  set  different  goals. 

d.  I  learned  things  to  improve  upon  the  first  time. 

e.  I  would  read  the  information  more  thoroughly. 

f.  I  would  try  to  be  a  smoother  talker. 

g.  I  would  know  what  others  wanted  from  me. 

Three  students  said  they  would  play  the  game  the  same 
way  the  second  time  because  they  thought  they  had  made 
the  best  possible  decision  under  the  circumstances. 

13.  Eighteen  participants  found  the  game  interesting  for  the 
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following  main  reasons: 

a.  We  had  never  done  this  sort  of  thing  before. 

b.  There  were  no  boring  parts  in  the  game. 

c.  The  game  provided  contact  with  others  and  was  a 
challenge. 

d.  This  was  much  different  from  what  we  had  done  in 
social. 

e.  This  game  showed  me  what  I  might  face  as  an  adult. 

f.  This  shows  what  people  involved  in  transition  must 
go  through. 

One  student  found  the  game  uninteresting  and  a  waste 
of  time. 

The  questionnaire  concluded  with  a  page  on  which 
students  were  asked  to  make  general  suggestions  for 
improving  the  game.  The  investigator  was  very  pleased 
with  the  number  of  responses.  The  following  were  the 
most  common  suggestions  offered: 

a.  Make  the  information-gathering  periods  longer  or 
have  more  of  them. 

b.  Explain  the  introduction  better  with  use  of  examples 
to  make  things  clearer  and  get  students  involved 
right  away. 

c.  Have  more  people  to  a  group  in  order  to  avoid 
arguments . 

d.  Have  more  people  to  a  group  to  make  for  better 
decisions. 

e.  Don't  have  any  job  cards.  In  real  life  you  would 
take  these  things  into  consideration  before  accepting 
a  job . 

f.  Improve  some  of  the  communities  and  have  their  good 
and  bad  points  more  equally  distributed. 

g.  Make  the  formulas  a  lot  simpler;  simplify  credits. 

h.  Have  more  people  and  more  cities  for  a  greater 
choice. 

i.  Have  hidden  cameras. 

j.  Give  more  power  to  the  administrators. 

k.  Have  an  extra  round  of  information  gathering  after  a 
final  decision  to  make  sure  that  everything  is  O.K. 


Worksheet  Summary 


Participant  worksheets  completed  on  the  first  day 
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indicated  that  the  following  were  the  major  goals  of  resident 
groups : 

1.  to  achieve  a  feasible  lifestyle  without  causing  any 
major  discomforts  (1) , 

2.  to  get  a  higher  income  (3)  , 

3.  to  have  a  better  job  and  a  better  place  to  live  with  a 
higher  income  (3), 

4.  to  get  a  quieter  living  town  with  the  same  standard  of 
living  (1) , 

5.  to  get  rid  of  debts  and  go  to  university  for  a  change  of 
atmosphere  (1) , 

6.  to  have  a  higher  standard  of  living  (1) , 

7.  to  find  a  good  clean  job  (1) , 

8.  to  get  more  of  an  outdoor  life  and  more  money  (1), 

9.  to  have  a  change  of  scenery  and  get  into  a  new  area  (1), 

10.  to  get  a  better  job  and  a  furthered  education  (1) . 

Worksheets  also  indicated  that  the  following  were 
the  specific  goals  of  resident  groups  participating  in 
Transition : 

1.  to  get  a  better  job  which  pays  more  (4)  , 

2.  to  educate  my  children  and  allow  them  to  grow  up  to  be 
well  adjusted  (1) , 

3.  to  get  closer  to  nature  and  the  outdoor  life  (3)  , 

4.  to  get  out  of  debt  (1) , 

5.  to  earn  more  money  for  a  better  quality  of  life  (1) , 

6.  to  get  good  sports,  cultural  and  social  facilities  (1), 
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7.  to  have  skiing  facilities  and  a  better  neighborhood  (2), 

8.  to  make  more  money  for  better  living  accommodations  and 
more  luxury  (1) . 

9.  We  want  a  more  secluded,  smaller  area  (1), 

10.  to  keep  the  family  together  (1) , 

11.  to  get  a  smaller,  cleaner  place  to  live  (1) , 

12.  to  get  into  a  higher  income  bracket  and  obtain  a  higher 
class  of  living  (2) . 

Worksheets  also  indicated  that  the  following  were 

the  major  goals  of  administrative  groups: 

1.  to  persuade  as  many  people  as  possible  to  live  and  stay 
in  our  community  (4) , 

2.  to  attract  worthy  people  to  reside  in  my  town  (1), 

3.  to  obtain  the  sixty  points  (1), 

4.  to  attract  as  many  semi-skilled,  skilled  and  professional 
people  to  our  community  as  possible  (2) . 

Worksheets  also  indicated  that  the  following  were 

the  personal  goals  of  administrative  groups: 

1.  to  see  if  I  am  as  persuasive  as  I  think  I  am  (1) , 

2.  to  promote  the  beauty  and  friendliness  in  my  town  to 
people  dissatisfied  in  their  present  location  (1) , 

3.  to  raise  the  quality  of  government  services  (1) , 

4.  to  assist  people  in  bettering  their  present  situation  (1). 

5.  We  would  like  to  tell  the  truth  and  nothing  but  the  truth 
about  our  community  (1) , 

6.  to  see  as  many  people  as  possible,  present  our  case  well 
and  convince  people  to  stay  here  (1) , 
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7.  to  satisfy  as  many  people's  needs  as  possible  (2). 

Analyses  of  Interviews ,  Questionnaire  and  Worksheets 

Administrative  groups  were  found  to  be  more 
open  than  resident  groups  in  both  the  interviews  and  the 
questionnaire.  This  phenomenon  may  be  attributable  to 
differences  in  general  roles  in  the  simulation.  Nevertheless 
it  would  appear  that  some  generalizations  can  be  made 
regarding  the  extent  of  cognitive  sharing  of  goals,  plans  and 
rules  among  participants  and  also  the  extent  of  affective 
sharing  of  reactions  to  their  experience  with  Transition.  An 
attempt  was  made  to  determine  the  extent  of  sharing  of  the 
grade  nine  students  in  real  life  decision-making  situations 
although  most  of  the  sharing  can  be  generalized  only  to  the 
hypothetical  decision-making  situations  into  which  Transition 
participants  were  placed.  All  generalizations  are  based  on 
informant  data  gleaned  from  the  interviews  with  twelve  parti¬ 
cipants,  the  twenty- three  returned  questionnaires  and  the 
worksheet  documents  completed  during  the  simulation  experience. 

While  there  was  no  evidence  to  indicate  complete 
cognitive  or  affective  sharing  among  all  participants,  the 
interviews  and  questionnaires  did  suggest  general  sharing  in 
the  following  areas: 

1.  Most  resident  groups  gave  high  priority  to  getting  a 
job  with  a  good  salary  as  one  of  their  major  or  specific 
goals.  Most  resident  groups  also  included  better 
recreation  facilities  in  their  goals  and  were  especially 
concerned  that  communities  provided  skiing  facilities. 
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2.  Administrators  generally  set  their  major  goal  as 
getting  as  many  people  as  possible  to  come  to  their 
community.  One  administrative  group,  however,  decided 
on  the  major  goal  of  informing  people  rather  than 
persuading  them.  They  wanted  a  few  totally  committed 
residents  rather  than  many  uncommitted  ones. 

3.  All  administrative  groups  viewed  introductory 
speeches  as  an  important  factor  in  their  success  in  goal 
achievement. 

4.  Resident  groups  generally  viewed  community  size  as 
insignificant  and  in  fact  often  made  final  decisions  in 
complete  ignorance  of  community  population  trends. 

5.  All  administrative  groups  perceived  the  deliberate 
selection  of  "good  points"  as  a  valid  way  to  sell  their 
communities . 

6.  Administrative  groups  were  generally  willing  to 
expend  their  ten  minute  interaction  rounds  with  a  limited 
number  of  resident  groups  if  they  so  desired. 

7.  All  administrative  groups  perceived  an  atmosphere  of 
competition  during  the  simulation  experience.  Varying 
degrees  of  role  internalization  appeared  to  account  for 
variations  in  the  extent  of  perceived  competition. 

8.  There  was  general  agreement  regarding  difficulty 
with  comprehension  of  formulas  and  point  systems  and  the 
introduction  and  explanation  of  Transition. 


9 .  There  was  general  agreement  that  role  descriptions 
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and  community  information  were  easy  to  understand  and 
that  making  a  final  decision  was  not  a  difficult  task. 

10.  The  great  majority  of  participants  felt  that  they 
would  play  their  roles  better  if  given  a  second  chance 
due  to  their  perception  of  having  learned  to  make  better 
decisions  as  a  result  of  their  simulation  experience. 

The  following  generalizations  can  be  made  regarding 
the  extent  of  affective  sharing  among  simulation  participants: 

1.  Groups  generally  equated  the  degree  of  satisfaction  and 
happiness  with  the  extent  to  which  they  achieved  their 
goals.  All  administrators  cited  helping  people  improve 
their  situations  as  the  greatest  source  of  satisfaction. 

2.  Groups  generally  found  the  simulation  experience  an 
enjoyable  one  and  "a  lot  of  fun"  whether  cognizant  or 
not  of  what  the  simulation  was  all  about. 

3.  The  great  majority  of  participants  viewed  their  role 
as  important  to  the  success  of  the  group  in  which  they 
were  working  and  the  entire  simulation. 

4.  Participants  generally  agreed  that  the  introduction 
turned  them  off  temporarily. 

5.  Most  participants  found  their  simulation  experience 
an  interesting  one.  Many  claimed  that  their  interest 
was  maintained  because  the  situation  was  a  novel  one  for 
their  social  studies  program  and  because  the  simulation 
provided  for  a  continuous  sharing  of  ideas  and  informa¬ 
tion  among  participants. 
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The  following  generalizations  can  be  made  regarding 
the  extent  of  cognitive  sharing  among  participants  in  their 
real  life  decision-making  situations: 

1.  Participants  generally  perceived  Canadians  as  giving 
most  consideration  to  job  and  salary  when  contemplating 
a  transition. 

2.  Participants  generally  agreed  that  such  important 
factors  as  social  and  cultural  activities  are  often 
ignored  by  those  making  a  transition. 

3.  Particpants  generally  agreed  that  both  factual 
information  and  people  influence  their  decision-making 
although  there  were  wide  variations  in  their  perceptions 
of  the  extent  of  influence  of  information  and  people. 

4.  Participants  generally  agreed  that  it  is  difficult 

to  find  the  perfect  job  that  gives  complete  satisfaction. 

5.  All  administrators  perceived  the  simulation  as  being 
able  to  help  them  in  future  decision-making.  They 
perceived  the  decision-making  model  as  helping  them 
examine  all  the  alternatives. 

6.  Participants  generally  agreed  that  the  best  decisions 
are  made  upon  an  examination  of  all  the  alternatives. 
There  was  disagreement  as  to  whether  or  not  they  always 
examine  all  the  alternatives  in  their  own  decision¬ 
making  . 

7.  There  was  no  sharing  of  an  understanding  of  the 
purpose  of  Transition  although  a  majority  did  feel  that 
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the  simulation  might  help  them  make  better  decisions  in 
the  future.  Approximately  half  of  the  participants 
described  Transition  as  showing  the  problems  of  decision¬ 
making  in  today's  world. 

8 .  Participants  varied  in  their  perceptions  of  how 
realistic  Transition  actually  was.  Some  felt  that  there 
was  too  much  of  a  chance  element  in  the  simulation  and 
not  enough  control  as  in  real  life. 

SUMMARY 

This  chapter  has  dealt  with  the  manner  in  which  a 
conceptual  scheme  of  formative  evaluation  became  operation¬ 
alized  during  the  developmental  stages  of  the  simulation 
Transition.  Four  phases  of  formative  evaluation  were 
discussed:  the  planning  or  preactive  phase,  two  interactive 

phases  during  which  students  at  Vernon  Barford  Junior  High 
School  transacted  with  the  simulation,  and  the  first  post¬ 
active  phase. 

In  the  preactive  phase,  important  decisions  were 
made  which  shaped  the  entire  evaluation.  It  was  first 
decided  that  an  investigation  of  the  process  dimension  of 
the  simulation  as  participants  engaged  in  decision-making 
would  be  the  most  logical  focus  for  formative  evaluation. 

The  ethnographic  research  approach  was  chosen  as  the  means 
of  formative  evaluation.  It  was  felt  that  such  an  approach 
would  provide  more  information  regarding  participant 
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transactions  and  interactions  than  the  traditional  statis¬ 
tical  approach  to  evaluation  and  would  catch  more  unantici¬ 
pated  outcomes  and  side-effects. 

Three  simulation  components  were  identified  as 
worthy  of  formative  evaluation:  the  simulation  model,  the 
sequence  of  activities  and  the  scenario  of  materials. 

Numerous  potential  feedback  sources  were  identified.  These 
included  direct  observations  of  the  simulation  leader  and 
non-participants,  audiotapes  and  videotapes  of  selected 
segments  of  simulation  activities,  post-simulation  inter¬ 
views,  questionnaire  and  classroom  documents. 

Section  2  of  this  chapter  has  focused  on  a  detailed 
account  of  the  first  interactive  phase  of  formative  evalu¬ 
ation,  an  initial  five  hour  testing  with  eleven  volunteer 
grade  nine  students  at  Vernon  Barford  School.  In  section  3 
has  been  presented  an  extensive  account  of  the  second  inter¬ 
active  phase  of  formative  evaluation,  a  major  class  piloting 
with  twenty-eight  grade  nine  students  also  at  Vernon  Barford 
School.  This  chapter  has  concluded  with  an  account  of  the 
first  postactive  phase,  an  analysis  of  ethnographic  data 
gathered  from  participant  interviews,  the  questionnaire  and 
participant  worksheet  documents. 
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CHAPTER  5 


SECOND  POSTACTIVE  PHASE: 

SIMULATION  ASSESSMENT  AND  REVISIONS 

The  role  of  this  formative  evaluation  study  was  the 
provision  of  product  information  into  the  developmental 
process  for  the  purpose  of  improving  the  product.  The 
evaluation  of  Transition  assumed  a  twofold  dimension  of 
description  and  judgement.  The  simulation  was  field  tested 
so  as  to  provide  decision-makers,  the  development  team,  with 
feedback  on  the  basis  of  which  revisions  could  be  produced. 

This  chapter  focuses  on  the  successes  and 
deficiencies  which  were  discovered  in  the  formalized 
evaluation  of  the  intermediate  version  of  Transition.  It 
deals  specifically  with  the  second  postactive  phase  of 
formative  evaluation,  the  assessment  and  subsequent  revisions 
of  the  three  fundamental  components  of  the  simulation:  the 
sequence  of  activities,  the  scenario  of  materials,  and  the 
basic  simulation  model  in  both  its  substantive  and  dynamic 
dimensions.  This  chapter  is  divided  into  the  following 
sections : 

1.  Transition:  Revision  of  Activities  and  Materials — 
Phase  1 

2.  Transition:  Revision  of  Activities  and  Materials — 
Phase  2 
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3.  Transition:  Assessment  of  the  Simulation  Model 

4 .  Summary 

TRANSITION:  REVISION  OF  ACTIVITIES  AND^  MATERIALS  — 

PHASE  1 

The  formal  process  of  revision  began  during  the  major 
piloting  at  Vernon  Barford  School.  On  the  third  day  of 
activities  simulation  participants  encountered  difficulties 
with  the  Satisfaction  Assessment  Worksheet.  Participants 
could  not  comprehend  the  procedure  for  self-assessment  and 
class  evaluation.  The  Fry  readability  formula  indicated 
that  the  worksheet  had  a  readability  index  of  "college" 
level.  The  worksheet  was  therefore  simplified  for  the  final 
day's  activities  during  which  time  participants  completed 
the  task  of  evaluation  more  adequately. 

The  first  revision  meeting  of  the  development  team 
was  conducted  one  week  after  the  major  piloting.  The 
exigency  of  this  meeting  was  due  to  the  decision  of  the  SURT 
team  to  have  non-developer  members  conduct  a  piloting  of 
Transition  with  their  grade  ten  and  eleven  classes  before 
the  end  of  the  school  year.  At  this  initial  meeting  emphasis 
was  thus  placed  on  the  identification  and  revision  of  major 
deficiencies  in  simulation  activities.  The  record  of 
observations  of  simulation  leader  and  non-participants  during 
the  major  piloting  served  as  a  point  of  reference  for  the 
identification  of  revisions. 
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The  following  initial  revisions  in  simulation 
activities  were  made: 

1.  The  importance  of  goal  setting  for  administrative 
groups  was  made  more  explicit.  The  major  piloting 
evidenced  lack  of  understanding  of  "personal  goals"  on 
the  part  of  administrators.  In  recognition  of  the 
importance  of  goal  setting,  it  was  decided  to  include 
examples  of  goals  in  administrative  group  manuals. 

2.  Decision  was  made  to  add  an  activity  in  which 
resident  groups  would  make  an  initial  class  presentation. 
Presentations  by  administrative  groups  had  appeared  to 
motivate  and  encourage  immediate  and  extensive  involve¬ 
ment  with  simulation  materials.  It  was  felt  that 
presentations  might  also  motivate  resident  groups  and 
lead  to  more  efficient  rounds  of  information  gathering. 
Resident  group  presentations  might  cue  administrators  as 
to  specific  resident  problems  that  would  be  discussed 
during  the  information  gathering  rounds.  It  was  decided 
that  resident  groups  should  not  reveal  their  goals  at 
this  initial  stage  of  the  simulation  as  this  might 
detract  from  subsequent  interactions. 

3.  Developers  were  cued  by  their  observations,  partici¬ 
pant  interviews  and  questionnaires  to  give  more  power  to 
administrative  groups.  During  the  major  piloting,  job 
and  housing  draw  cards  were  placed  on  a  large  map  at  the 
front  of  the  classroom.  It  was  found  that  administrative 
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groups  had  no  control  over  who  received  their  available 
jobs  and  living  accommodations.  It  was  decided  to  give 
administrators  the  control  of  resident  qualifications 
for  jobs  and  housing  and  the  draw  cards.  In  conjunction 
with  the  allocation  of  more  power  to  administrators,  it 
was  decided  to  structure  their  satisfaction  assessment 
activity  and  final  class  presentation  so  that  they  would 
have  to  justify  their  decisions  regarding  the  awarding 
of  jobs  and  living  accommodations. 

4.  All  formulas  were  simplified  and  clarified.  The 
formula  for  determining  variations  in  housing  costs  was 
simplified  and  activities  re-sequenced  so  that  credit 
points  would  be  determined  only  after  the  draw  of  housing 
and  job  cards.  Cost  of  moving  formulas  were  completely 
eliminated  and  replaced  by  the  addition  to  administrative 
booklets  of  a  table  of  designated  moving  costs.  An 
explanation  of  the  conversion  factor  was  included  in 
participant  manuals  and  procedures  to  determine  variations 
on  living  costs  were  simplified.  Rules  were  revised  to 
specify  that  resident  groups  were  obligated  to  remain 
within  their  job  classification.  This  revision  was 
deemed  crucial  in  order  that  job  alternatives  and  job 
placement  be  realistic. 

5.  Decision  was  made  to  structure  all  final  group 
presentations.  The  major  piloting  evidenced  a  need  for 
participant  group  resumes  to  be  substantiated  so  as  to 
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make  class  evaluations  of  each  group’s  performance  more 
meaningful.  It  was  decided  to  include  in  participant 
manuals  a  description  of  the  final  evaluation  activity 
so  that  participants  would  be  cognizant  of  their 
responsibilities  during  final  presentations  from  the 
outset  of  the  simulation. 

6.  Observations  of  the  developers  during  the  major 
piloting  agreed  that  those  groups  who  waited  to  gather 
all  their  information  from  all  communities  before  making 
a  final  decision  appeared  to  handle  their  roles  best. 
Nevertheless  it  was  decided  not  to  make  one  approach  to 
decision-making  mandatory.  It  was  recognized  that  the 
simulation  should  not  be  so  structured  as  to  severely 
restrict  freedom  of  participants.  Alternative  ways  of 
handling  situations  were  deliberately  left  open  so  that 
all  participant  groups  would  be  able  to  decide  on  what¬ 
ever  means  they  wished  to  achieve  their  goals. 

Participant  manuals  and  information  packages 
indicating  the  above  revisions  were  retyped,  collated  and 
compiled  for  the  additional  class  pilotings. 

TRANSITION:  REVISIONS  OF  ACTIVITIES  AND  MATERIALS — 

PHASE  2 

A  second  series  of  revision  meetings  was  conducted 


one  month  after  the  major  piloting.  The  following  served  as 
points  of  reference  for  the  development  team  in  their  further 
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identification  of  revisions: 

1.  Findings  of  the  participant  questionnaire. 

2.  Transcripts  of  participant  interviews. 

3.  Audiotapes  and  videotapes  of  selected  segments  of 
activities . 

4.  Observations  of  simulation  leader  and  non-participants 
during  the  major  piloting. 

5.  Observations  of  non- developer  simulation  leaders 
during  the  second  series  of  class  pilotings. 

6.  Observations  of  an  external  evaluator  of  SURT 
materials  during  the  second  series  of  class  pilotings. 

The  following  revisions  of  simulation  materials  and 
activities  were  made  at  this  stage  of  development: 

1.  All  simulation  materials  were  simplified  and 
clarified.  In  the  questionnaires  participants  reported 
that  they  were  "turned  off"  by  segments  of  the  partici¬ 
pant  manual  which  were  lengthy  and  difficult  to 
comprehend.  Some  participants  observed  that  scenarios 
were  written  in  a  'university'  style.  The  Fry  read¬ 
ability  formula  suggested  that  segments  of  materials  had 
a  readability  level  of  "grade  twelve".  A  concerted 
effort  was  made  to  make  materials  more  readable  for 
junior  and  senior  high  school  students.  Grammatical 
errors  and  cumbersome  wording  were  corrected.  Paragraphs 
were  shortened  in  the  participant  manual  and  information 
booklets.  The  explanation  of  rules  and  procedures  for 
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goal  achievement  was  simplified. 

2.  A  major  revision  was  made  in  the  participant  manual. 
Observations  of  the  simulation  leader  and  non-participants 
during  the  major  piloting  suggested  lack  of  participant 
understanding  of  the  relationship  between  major  and 
intermediary  goals.  Administrative  groups  in  particular 
experienced  difficulties  in  setting  intermediary  goals. 

The  relationship  between  major  and  intermediary  goals 

was  made  more  explicit  in  the  manual.  The  terms  "specific 
goals"  and  "personal  goals"  were  changed  to  "intermediary 
goals".  A.  goal  setting  model  and  illustrations  of  how 
both  resident  and  administrative  groups  might  apply  the 
model  were  added  to  the  manual  in  an  attempt  to  clarify 
the  process  of  goal  setting  and  the  relationship  between 
the  two  types  of  goals.  The  illustrations  emphasized  the 
need  for  intermediary  goals  to  be  attained  before  major 
goals  could  be  achieved.  The  manual's  explanation  of 
goal  setting  was  revised  to  point  out  that  goals 
should  be  re-evaluated  throughout  the  simulation  and 
could  be  revised  on  the  basis  of  knowledge  and  experience 
gained  subsequent  to  initial  goal  setting. 

3.  Analysis  of  participant  questionnaires  and  observa¬ 
tions  of  non-developer  simulation  leaders  during  the 
second  major  piloting  suggested  occasional  inactivity 
and  boredom  on  the  part  of  administrative  groups.  It 
was  decided  therefore  to  structure  the  activities  of 
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administrators.  Role  profiles  were  drawn  up  for  adminis¬ 
trative  groups.  Worksheets  similar  to  those  of  resident 
groups  were  developed  so  as  to  structure  administrators' 
designative  inquiry  into  their  community  situations  and 
appraisive  inquiry  in  goal  setting. 

4.  The  first  major  piloting  evidenced  participant 
confusion  regarding  simulation  resources.  The  develop¬ 
ment  team  was  especially  concerned  with  administrative 
group  inability  to  gain  satisfaction  points.  The 
explanation  of  the  point  system  was  thus  simplified  to 
include  concise  definitions  of  credit  and  satisfaction 
points  and  a  step  by  step  account  of  how  points  could  be 
attained.  In  recognition  of  the  affective  dimension  of 
decision-making,  the  point  system  was  revised  so  that 
satisfaction  points  were  directly  related  to  goal  achieve¬ 
ment.  A  simplified  procedure  for  the  calculation  of 
satisfaction  points  was  included  in  the  participant 
manual  along  with  a  description  of  points  being  directly 
based  on  the  extent  of  satisfaction  with  goal  achievement 
and  with  general  playing  of  the  role . 

5.  All  piloting  of  Transition  indicated  that  resident 
group  worksheets  were  extremely  ambiguous  and  confusing. 
The  fact  that  participants  took  more  than  an  hour  to 
complete  a  scheduled  twenty-five  minute  activity  was  an 
obvious  cue  for  revision.  The  worksheet  was  thus  revised 
to  include  the  following: 
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a.  a  re-wording  of  all  designative  questions, 

b.  a  simplified  rating  of  community  facilities 
and  opportunities, 

c.  specification  of  where  information  for  the 
answering  of  questions  could  be  found,  and 

d.  suggestion  for  participants  to  assume  the  role 
and  impose  their  own  value  systems. 

TRANSITION:  ASSESSMENT  OF  THE  SIMULATION  MODEL 

An  assessment  of  the  Transition  model  involved  an 
evaluation  of  the  substantive  model  of  the  simulation,  the 
SURT  Community  Model,  and  also  the  dynamic  model  of  decision¬ 
making  as  presented  by  Morris'  three  stages  of  inquiry: 
designative,  appraisive  and  prescriptive  inquiry. 

The  development  team's  observations  of  the  substan¬ 
tive  process  of  decision-making  during  the  simulation 
experience  suggested  that  participants  were  not  constrained 
by  the  focus  on  four  systems — economic,  educational, 
recreational  and  governmental — in  Transition .  Nor  was  there 
any  indication  that  other  subsystems  in  the  Community  Model 
were  completely  excluded  in  the  simulation.  Participants 
appeared  able  to  impose  their  own  beliefs  and  values  as  well 
as  those  of  the  role  they  were  playing.  In  some  cases,  cul¬ 
tural  ties,  family  ties  and  religious  beliefs  and  values  had 
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a  definite  influence  on  the  decisions  which  were  made.  This 
was  interpreted  as  a  cue  that  the  SURT  Community  Model  upon 
which  Transition  is  substantively  based  has  the  integrity  of 
a  system.  The  systemic  model  seemed  to  allow  for  a  flow  of 
actual  and  potential  data  in  a  manner  which  approximates  the 
community  systems  being  simulated  in  Transition. 

The  fact  that  resident  and  administrative  groups  were 
able  to  impose  their  own  value  systems  was  evidence  that, 
although  the  focus  in  Transition  was  on  four  subsystems, 
other  subsystems  including  the  religious,  cultural  and  family 
subsystems  were,  at  least  tangentially,  incorporated  into  the 
substantive  model  of  the  simulation.  During  the  formal 
revision  stage  it  was  decided  to  include  a  simplified 
explanation  of  the  SURT  Community  Model  in  both  simulation 
leader  and  participant  manuals. 

While  this  investigation  was  concerned  in  part  with 
the  integrity  of  the  substantive  model  of  the  simulation,  a 
concern  of  equal  if  not  more  importance  was  whether  or  not 
participants  in  Transition  went  beyond  the  study  of  the 
substantive  nature  of  community  to  an  actual  involvement  in 
the  dynamic  process  of  decision-making.  Therefore,  an 
attempt  was  made  to  determine  the  inquiry  process  of 
participants  as  they  engaged  in  the  solution  of  problems 
associated  with  transition  from  one  community  to  another. 

Observations  of  the  simulation  leader  and  non¬ 
participants  gave  some  indication  that  participants  were 
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using  the  method  of  inquiry  incorporated  in  the  simulation 
in  a  discernable  way  without  having  any  formal  introduction 
to  it.  During  final  debriefing  sessions  all  participants 
were  formally  presented  with  a  simplified  version  of  the 
inquiry  model.  Volunteer  students  proved  quite  capable  of 
an  explanation  of  how  they  had  played  their  role  in  terms  of 
designative,  appraisive  and  prescriptive  inquiry. 

Analysis  of  the  videotape  of  main  simulation 
activities  suggested  that  the  inquiry  process  at  the  macro 
level,  the  class  as  a  whole,  was  predominantly  designative 
at  the  outset  of  the  simulation.  Both  resident  and  adminis¬ 
trative  groups  acquainted  themselves  with  specified  roles 
and  attempted  to  answer  the  basic  designative  question  "What 
is  my  present  situation?"  Groups  generally  then  engaged  in 
goal  setting  and  attempted  to  answer  the  basic  appraisive 
question  "What  do  I  want?"  Participant  groups  were  eventually 
led  to  a  consideration  of  what  action  to  accord  preference 
as  they  attempted  to  answer  the  predominantly  prescriptive 
question,  "What  should  I  do  to  get  what  I  want?"  Throughout 
all  simulation  activities  there  appeared  an  interplay  of 
designative,  appraisive  and  prescriptive  inquiry  as  partici¬ 
pants  were  directed  to  an  attempt  to  solve  the  basic  problems 
of  how  to  make  successful  transition. 

Analysis  of  audiotapes  of  interactions  within  each 
group  indicated  a  general  goodness  of  fit  between  the  inquiry 
process  at  the  macro  level  and  that  at  micro  levels.  For 
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most  participant  groups  the  setting  of  a  major  goal  posed 
problems  of  how  to  achieve  that  goal.  Most  groups  deliberated 
over  the  alternative  courses  of  action  which  could  best 
achieve  their  intermediary  and  ultimate  major  goal  before 
making  a  final  decision.  However,  there  were  exceptions. 

Some  resident  groups  made  decisions  without  designative 
inquiry  into  jobs,  housing  and  services  available  in  all 
five  communities.  Others  made  decisions  without  undergoing 
the  structured  activity  of  setting  both  intermediary  and 
major  goals  and  re-evaluating  their  goals.  Some  groups 
definitely  engaged  in  decision-making  in  a  more  unstructured 
way  than  others.  The  fact  that  the  decision-making  model 
accommodated  groups  who  deviated  from  the  structured  inquiry 
process  was  interpreted  as  a  cue  that  the  'process'  model, 
while  structured  to  an  extent,  was  nevertheless  flexible 
enough  to  give  participants  freedom  to  make  decisions  in 
structured  and  unstructured  ways. 

Subsequent  interviews  with  administrative  groups 
indicated  that  some  participants  viewed  the  decision-making 
model  of  Transition  as  a  logical  one  but  one  not  always  used. 
This  observation  did  suggest  that  participants  were  engaged 
in  the  dynamic  process  of  decision-making  beyond  the  mere 
systemic  or  substantive  process.  While  the  investigation 
did  not  attempt  to  measure  what  students  learned,  analysis 
of  interview  transcripts  and  questionnaires  suggested  that 
some  participants  recognized  for  the  first  time  the  complex 
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dimensions  of  decision-making  regarding  transition.  Some 
interviewed  participants  indicated  that  the  simulation  had 
taught  them  to  examine  alternatives  more  carefully  before 
making  decisions. 

There  was  no  evidence  to  suggest  that  participants 
had  revised  their  goals  at  any  stage  in  the  simulation 
although  numerous  opportunities  for  re-evaluation  and  revision 
were  provided  during  the  first  major  piloting.  Nevertheless, 
it  appeared  that  all  participants  were  engaged  to  a  greater 
or  lesser  extent  in  the  dynamic  problem  solving  process.  It 
was  concluded  that  the  decision-making  model  was  valid  since 
most  participants  engaged  in  designative,  appraisive  and 
prescriptive  inquiry  in  a  discernable  way  and  found  the 
inquiry  model  helpful  in  decision-making.  The  model  was  also 
flexible  enough  to  accommodate  those  who  did  not  use  a 
structured  inquiry  process  to  make  decisions  and  who  clearly 
revealed  that  decisions  are  not  always  made  in  a  logical 
structured  way. 

Throughout  the  developmental  stages  of  Transition , 
materials  were  continually  revised  to  present  participants 
with  problems  to  be  solved  and  to  have  greater  potential  for 
in-depth  student  involvement.  Activities  were  continually 
re-structured  and  re-sequenced  so  as  to  allow  both  residents 
and  administrative  groups  opportunities  to  solve  their 
problems  via  involvement  in  the  dynamic  process  of  decision¬ 
making.  A  concerted  effort  was  made  to  ensure  that  partici¬ 
pants  in  Transition  would  engage  in  designative,  appraisive 
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and  prescriptive  inquiry.  Such  a  process  of  formative 
evaluation  of  the  basic  decision-making  model  throughout 
the  development  process  almost  guaranteed  that  few  major 
revisions  of  the  model  would  be  required. 

SUMMARY 

In  this  chapter  has  been  presented  a  detailed 
account  of  the  actual  assessment  and  subsequent  revisions  of 
the  three  basic  components  of  Transition :  the  sequence  of 
activities,  the  scenario  of  materials  and  the  simulation 
model  in  terms  of  its  substantive  and  decision-making 
dimensions.  The  formal  revisions  began  during  the  major 
class  piloting  and  continuted  for  a  month  and  a  half 
thereafter.  In  this  Chapter  has  been  described  the 

manner  in  which  numerous  data  sources  influenced  the 
assessment  and  revisions  made  by  the  decision-makers,  the 
simulation  development  team. 
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CHAPTER  6 


SUMMARY ,  CONCLUSIONS  AND  RECOMMENDATIONS 

The  purpose  of  this  study  was  to  develop  and  opera¬ 
tionalize  a  conceptual  scheme  for  formative  evaluation  of  a 
simulation  called  Transition.  Transition  was  designed  for 
Canadian  students  of  secondary  school  social  studies  as  part 
of  the  curriculum  package  of  Project  SURT,  a  subproject  of 
Project  Canada  West.  The  study  was  specifically  intended  to 
provide  feedback  information  into  the  developmental  process 
in  order  that  decision-makers,  the  simulation  development 
team,  could  improve  the  product. 

The  study  assumed  an  essentially  two-fold  dimension: 
a  descriptive  dimension  and  a  judgemental  dimension.  An 
attempt  was  made  to  describe  and  assess  the  transaction 
between  participant  students  and  the  simulation.  The  study 
was  viewed  as  an  initial  investigation  into  the  process 
dimension  of  a  simulation. 

The  formative  evaluation  was  guided  by  three 
questions : 

1.  What  inquiry  process  do  participants  use  in  handling 
information  presented  them  by  Transition  and  how  well  do 
they  use  the  structured  inquiry  process  of  the  simulation? 

2.  What  activities  are  participants  engaged  in  during  their 
simulation  experience  and  how  well  do  they  engage  in  the 
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identified  activities  of  Transition? 

3.  What  are  participant  perceptions  and  attitudes  to  their 
simulation  experience? 

Literature  and  research  relating  to  three  themes  was 
reviewed  for  this  study:  formative  evaluation  during  program 
development,  trends  in  simulation  evaluation  and  the  ethno¬ 
graphic  approach  to  evaluative  research.  Literature  relating 
to  formative  evaluation  provided  the  basis  for  the  conceptual 
scheme  for  the  evaluation  of  Transition  which  was  developed. 
The  writings  of  Scriven  (1967)  gave  particular  insight  into 
the  goals  and  roles  of  formative  evaluation  as  distinct  from 
summative  evaluation. 

Three  major  reviews  of  simulation  research: 
Cherryholmes  (1966),  Fletcher  (1971)  and  Wentworth  and  Lewis 
(1973)  indicated  past  trends  in  simulation  evaluation  and 
pointed  to  the  exigency  of  future  evaluations  to  focus  on 
the  process  rather  than  the  learning  effects  of  simulations. 

A  review  of  selected  simulation  studies  indicated  that 
evaluators  have  more  often  than  not  probed  questions  of 
convenience  rather  than  those  of  importance  to  simulation 
developers  and  users.  Literature  related  to  simulations 
cued  the  investigator  of  the  need  to  develop  a  broad  enough 
research  paradigm  to  allow  for  an  analysis  and  evaluation  of 
the  way  Transition  might  affect  the  total  environment  of  the 


classroom. 

Literature  related  to  the  ethnographic  approach  to 
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evaluation  pointed  out  an  alternative  to  the  statistical 
orientation.  An  ethnographic  orientation  was  chosen  for 
this  study  as  a  means  of  formative  evaluation  which  might 
indicate  more  about  the  complex  transactions  and  interactions 
of  the  instructional  setting.  The  literature  and  research 
reviewed  provided  many  cues  as  to  how  to  develop  an  opera¬ 
tional  scheme  for  the  formative  evaluation  of  Transition. 

A  description  of  the  stages  of  development  of 
Transition  was  presented  in  Chapter  3.  The  Johnson-Aoki 
Curriculum  Development  Model  provided  the  conceptual  frame¬ 
work  for  the  development  of  the  simulation.  Both  a 
systemic  model  and  a  problem-solving  model  were  incor¬ 
porated  into  the  simulation  model.  The  SURT  Community 
Systems  Model  was  chosen  as  the  substantive  model  for 
Transition .  The  method  of  inquiry  postulated  by  Morris  was 
chosen  as  the  theoretical  base  for  the  decision-making  model 
incorporated  as  the  core  of  the  simulation. 

The  development  of  the  intermediate  versions  of 
Transition  was  in  many  ways  a  non-linear  process.  The 
inquiry  process  of  the  development  team  was  often  a  most 
unstructured  one.  Specific  objectives  for  Transition  were 
not  formulated  until  after  some  three  months  of  exploration 
of  ideas  and  materials.  The  actual  process  of  formative 
evaluation  began  with  the  first  meeting  of  the  simulation 
development  team  and  became  an  integral  part  of  the  entire 
developmental  process. 
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Four  phases  of  operationalizing  the  conceptual  scheme 
for  formative  evaluation  were  discussed  in  Chapter  4.  In  the 
preactive  phase,  decision  was  made  to  focus  on  the  process 
dimension  of  Transition  rather  than  engage  in  a  pretest- 
posttest  study.  In  order  to  capture  key  transactions  and 
interactions,  it  was  decided  to  audiotape  and  videotape 
selected  segments  of  the  simulation  experience.  Observations 
of  simulation  leader  and  non-participants,  participant  inter¬ 
views,  a  questionnaire  and  classroom  documents  were  also 
recognized  as  important  potential  data  sources  for  the 
formalized  evaluation  study.  Three  components  of  the 
simulation  were  identified  as  worthy  of  formative  evaluation: 
the  sequence  of  activities,  the  scenario  of  materials  and  the 
simulation  model  in  both  its  substantive  and  process 
dimensions . 

The  first  interactive  phase  of  formative  evaluation 
involved  an  initial  testing  with  eleven  volunteer  grade  nine 
students  at  Vernon  Barford  Junior  High  School.  The  second 
interactive  phase  consisted  of  a  major  piloting  with  a  grade 
nine  social  studies  class  of  twenty-eight  students.  This 
piloting  lasted  for  seven  hours  over  four  teaching  days. 

The  simulation  leader  for  the  piloting  was  the  classroom 
teacher  and  co-developer  of  the  simulation,  Jane  Sterk. 

Both  initial  testing  and  major  piloting  were  observed  in 
their  entirety  by  the  investigator. 

The  first  postactive  phase  of  the  evaluation  was 
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reported  in  the  concluding  section  of  Chapter  4.  During  this 
phase  all  data  gathered  during  the  first  and  second  inter¬ 
active  phases,  the  actual  student  transactions  with  the 
simulation,  were  analyzed  from  an  ethnographic  perspective. 

The  assessment  and  subsequent  revisions  of  the  model, 
activities  and  materials  of  Transition  were  reported  in 
Chapter  5.  An  attempt  was  made  to  describe  the  manner  in 
which  numerous  ethnographic  data  sources  influenced  the 
decisions  of  the  development  team  at  this  stage  of  the 
developmental  process. 

The  revised  edition  of  Transition  will  be  retyped, 
collated  and  compiled  for  major  piloting  in  thirty  classrooms 
across  Canada  in  1974-75  as  part  of  the  SURT  curriculum 
development  package.  Some  questions  regarding  Transition 
which  remain  unanswered  at  the  present  time  include: 

1.  For  what  level  of  the  secondary  school  social  studies 
curricula  is  Transition  best  designed? 

2.  For  what  curricula  other  than  social  studies  is 
Transition  suitable? 

3.  How  do  observations  of  non-developer  simulation  leaders 
across  Canada  regarding  successes  and  deficiencies  of 
Transition  compare  with  those  of  the  developers  during 
the  first  major  piloting? 

How  do  the  perceptions  and  attitudes  of  students  across 
Canada  compare  with  those  of  the  students  who  partici¬ 
pated  in  the  first  major  piloting? 
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5.  Does  Transition  involve  students  in  the  dynamics  of 
decision-making  when  piloted  in  classrooms  of  non¬ 
developers? 

6.  Do  systematic  evaluations  of  Transition  carried  out  in 
the  1974-75  school  year  produce  significant  findings 
regarding  the  simulation  model,  activities  and  materials 
which  the  formative  evaluation  overlooked? 

There  are  many  general  questions  in  the  mind  of  the 

investigator  for  which  future  evaluative  research  might 

provide  answers.  These  include  the  following: 

1.  Is  an  investigation  of  the  transaction  process  a  logical 
and  worthwhile  focus  for  an  evaluation  of  forms  of 
instructional  content  other  than  simulations? 

2.  Is  the  ethnographic  approach  more  appropriate  for  an 
analysis  and  evaluation  of  real  life  situations  than  of 
hypothetical  situations  such  as  are  presented  by 
simulations? 

3.  Are  there  limitations  in  the  ethnographic  approach  to 
evaluations  conducted  in  classroom  settings? 

4.  What  effects  do  various  methods  of  introducing  instruc¬ 
tional  content  and  various  methods  of  grouping  students 
have  on  subsequent  transactions  with  content  displayed? 

5.  Why  do  some  displays  of  instructional  content  involve 
students  while  others  do  not?  In  what  way  can  the 
potential  of  instructional  content  for  in-depth  student 
involvement  be  maximized? 
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6.  Do  specific  simulations  on  the  education  marketplace 
provide  situations  that  enable  students  to  make 
decisions? 

7.  Can  goals  and  roles  of  summative  evaluation  be  incor¬ 
porated  into  systematic  evaluation  of  instructional 
content  and  teaching  strategies  in  their  formative 
stages  of  development? 

CONCLUSIONS 

This  study  was  of  an  exploratory  nature  into  the 
process  dimension  of  a  simulation  in  its  developmental  stages 
by  means  of  an  ethnographic  research  approach.  The  following 
conclusions  have  been  drawn  from  the  study: 

1.  The  ethnographic  method  has  been  found  to  be  a  most 
useful  and  fitting  orientation  for  formative  evaluation 
of  instructional  content  and  teaching  strategies  incor¬ 
porated  in  a  simulation.  The  ethnographic  approach  has 
provided  the  investigator  with  a  deeper  knowledge  about 
the  interrelations  among  teacher,  students  and  their 
environment  within  the  context  of  a  simulation.  The 
adoption  of  an  ethnographic  orientation  to  evaluation 
facilitated  a  holistic  investigation  of  the  classroom 
transaction  process  and  allowed  the  investigator  to  be 
immersed  in  direct  contact  with  student  participants  in 
the  simulation. 

The  ethnographic  approach  proved  most  conducive  to 
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an  examination  of  the  process  dimension  of  the  simulation, 
permitting  the  investigator  to  conduct  research  in  an 
unstructured  way  and  to  be  particularly  open  to  search  for 
unanticipated  teacher  and  student  reactions  to  the  simulation. 
The  ethnographic  method  should  not  be  used  when  a  structured 
research  approach  is  desirable  or  when  rigorous  designs  and 
objectivity  are  essential  to  an  evaluative  research  study. 

2.  This  study  has  convinced  the  investigator  of  the 
crucial  complementarity  of  the  process  of  development 
and  formative  evaluation.  Designers  of  instructional 
content  have  too  often  indicated  a  need  for  evaluation 
but  have  merely  appended  evaluation  to  the  developmental 
process.  The  investigator  feels  strongly  that  developers 
must  not  be  so  product  oriented  as  to  ignore  the  importance 
of  formative  evaluation.  They  must  consider  development 
and  formative  evaluation  as  integral  and  parallel  stages 
in  the  developmental  process.  Formative  evaluation  is 

too  crucial  to  be  considered  as  an  afterthought. 

3.  The  evaluation  which  was  built  into  the  developmental 
stages  in  Transition  involved  very  careful  planning  and 

a  considerable  investment  of  time  and  energy.  Only 
after  the  study  was  the  investigator  fully  convinced  that 
such  an  investment  was  worthwhile.  Any  thorough  attempts 
by  curriculum  developers  and  instructional  planners  to 
improve  their  products  can  be  most  enervating.  However, 
the  payoff  is  surely  to  be  an  improvement  in  the  quality 
of  instructional  content  and  strategies  of  teaching  in 
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our  schools. 

4.  The  decision  to  focus  on  the  quality  of  the 
existential  situation  during  student  transaction  with 
content  displayed  proved  to  be  a  most  wise  one.  The 
investigation  into  intended  and  unanticipated  experien¬ 
tial  outcomes  was  found  to  be  well  suited  to  the  role  of 
formative  evaluation,  possibly  moreso  than  an  investiga¬ 
tion  of  terminal  learning  outcomes  might  have  been. 

5.  While  summative  evaluation  has  a  role  to  play  in  the 
developmental  process,  the  author  feels  that  formative 
evaluation,  though  often  underestimated  if  not  completely 
ignored,  has  an  equal  if  not  more  important  role.  More 
meaningful  and  extensive  revisions  to  a  product  can  be 
made  while  that  product  is  in  a  fluid  stage  of  develop¬ 
ment.  In  view  of  the  fact  that  external  non-developers 
played  such  an  important  role  in  the  formative  evaluation 
of  Transition,  the  author  feels  that  the  traditional 
goals  and  roles  of  summative  evaluation  might  be  able  to 
be  incorporated  into  systematic  evaluation  of  instruc¬ 
tional  content  and  teaching  strategies  in  their  formative 
stages  of  development. 

RECOMMENDATIONS  FOR  FURTHER  STUDY 

The  following  recommendations  can  be  made  for  future 
researchers : 


1. 


It  is  recommended  that  formative  evaluations  of  forms 
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of  instructional  content  other  than  simulations  be 
conducted  by  future  researchers.  It  is  suggested  that 
consideration  be  given  to  the  ethnographic  approach  as  a 
most  fitting  orientation  for  formative  evaluation. 

2.  It  is  suggested  that  more  extensive  classroom  ethno¬ 
graphies  be  conducted  over  extended  periods  of  time. 

These  might  provide  a  description  of  rules  and  plans 
which  students  use  to  achieve  intended  outcomes  in  real 
life  situations  rather  than  hypothetical  situations  as 
presented  by  simulations.  More  extensive  investigations 
into  classroom  transactions  via  the  ethnographic  approach 
might  lead  to  the  identification  of  important  instruc¬ 
tional  variables  and  their  antecedents  and  consequences 
overlooked  by  transactional  analyses  to  date. 

3.  It  is  recommended  that  the  transaction  process  be 
the  focus  for  future  formative  and  summative  evaluation 
studies.  Such  studies  should  explore  the  use  of 
carefully  planned  videotapes  and  audiotapes  of  selected 
segments  of  the  classroom  transactions,  extensive  inter¬ 
views  with  students  during  and  after  the  transaction 
process,  student-developed  materials  and  recorded  direct 
observations  of  participant  teachers  and  non¬ 
participants  as  potential  means  of  tapping  feedback  from 
students  and  gaining  insight  into  their  inside  point  of 
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APPENDIX  A 


PARTICIPANT-NON  PARTICIPANT  OBSERVATION  SHEET 
Directions 


Using  the  designated  Time  Line  of  Simulation 
Activities  as  a  guide,  please  describe  each  activity 
specifically  as  you  witnessed  it  happen  in  terms  of  the 
simulation  leader  activities,  student  transactions  and 
interactions,  and  particular  simulation  materials.  Also 
indicate  your  judgement  as  to  the  successes  and  deficiencies 
as  each  activity  was  carried  out. 


Activity  #1  (1) 

Simulation  Leader: 

A  little  disorganized  on  part  of  game  master.  Some 
student  interaction.  However  mainly  of  the  question-answer 
variety.  Students  seemed  a  bit  subdued  because  of  mikes  and 
camera  equipment. 

2nd  Developer: 

Jane  explained  the  development  of  Transition. 

Activity  adequate. 

Investigator: 

Leader  in  control  of  class.  Presented  information 
to  students  re  Canada  Studies,  P.C.W.  and  SURT.  Students 
listened  attentively  for  whole  activity,  except  for  one 
group  of  boys .  Background  information  appeared  to  be 
occasionally  going  over  heads  of  students. 

Activity  #1  (2) 

Simulation  Leader: 

Cannot  distinguish  this  as  separate  from  above.  I 
don't  think  the  idea  of  formative  evaluation  was  well  enough 
conveyed.  No  student  interaction.  Simply  giving  information. 

2nd  Developer: 

Jane  explained  the  role  of  students  in  formative 
evaluation  of  Transition.  Don't  really  feel  it  mattered 
what  was  said.  Possibly  having  had  students  write  down 
their  problems  and  suggestions  regarding  the  game  may  have 
made  this  part  more  meaningful. 
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Investigator : 

Leader  indicated  the  importance  of  student  roles  in 
the  revisions  of  the  simulation.  This  appeared  to  be  a  most 
important  activitiy  for  it  seemed  to  encourage  students  to 
co-operate  in  subsequent  activities.  Students  listened 
attentively. 

Activity  #1  (3) 

Simulation  Leader: 

Comaprison  of  a  simulation  with  an  athletic  game 
seemed  to  work  well.  Students  were  quiet  but  volunteered 
answers  when  questions  were  asked. 

2nd  Developer: 

A  few  questions  were  raised.  I  think  the  presenta¬ 
tion  was  good  for  students  not  having  some  knowledge  of 
simulations.  Others  should  be  given  general  information 
about  simulations  prior  to  starting  the  game. 

Investigator : 

Leader  should  reiterate  student  comments  so  that  the 
whole  class  can  hear  and  benefit  from  ideas  and  suggestions. 
The  transparency  on  the  relationship  between  simulations  and 
athletic  games  was  very  good  in  maintaining  attention,  as 
did  reference  to  the  Edmonton  Eskimos.  Boys  immediately 
perked  up. 

Activity  #2  (1) 


Leader: 


It  would  have  helped  to  have  a  pointer  when  showing 
areas  and  centres  involved.  Description  of  fictional  country 
seemed  adequate  although  some  questions  wdre  asked  about 
latitude.  It  is  important  to  indicate  what  sides  are  bound 
by  water  and  what  ones  by  land.  Also,  description  of  Lake 
Mistehae  was  omitted. 

2nd  Developer: 

Description  of  provinces  and  communities  being 
simulated  very  well  presented. 

Investigator: 

It  would  be  wise  for  the  leader  to  use  a  pointer 
and  refer  directly  to  the  map.  Information  presented 
clearly  and  explicitly. 
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Activity  #2  (2) 
Leader: 


This  may  have  been  a  little  skimpy.  Types  of  roles 
might  have  been  clarified  a  bit  more.  Some  students 
indicated  great  confusion  at  the  end  of  this  session.  No 
student  interaction. 

2nd  Developer: 

Oral  presentation  quite  good.  Blackboard  outline  or 
one  on  overhead  would  have  provided  a  more  adequate  frame  of 
reference . 

Investigator : 

Clearer  explanation  needs  to  be  given  of  the  role  of 
administrators,  as  well  as  the  fact  that  each  group  plays 
only  one  role.  Students  appeared  interested  and  attentive 
but  quiet. 

Activity  #2  (3) 

Leader: 


Fairly  orderly.  A  few  questions  as  students  went 
through.  Students  were  quiet  and  appeared  to  concentrate  on 
what  was  being  read.  Took  longer  than  15  minutes  for 
students  to  complete  booklet.  What  can  be  done  about 
students  who  don't  read  carefully?  Some  students  tried  to 
help  others. 

2nd  Developer: 

Distribution  and  reading  appeared  to  be  effectively 
carried  out. 

Investigator : 

Very  orderly  procedure  for  passing  out  manuals.  Little 
or  no  interaction  at  this  point.  Each  student  read  the 
booklet  himself. 

Activity  #2  (4) 

Leader : 

It  appeared  that  students  didn't  have  many  questions 
as  game  master  went  through  manual,  however  this  will  be 
seen  later.  Some  time  should  have  been  spent  on  the  point 
system — what  the  points  were  and  how  to  gain  points. 
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2nd  Developer: 

Few  questions  since  few  students  actively  transacted 
with  materials  at  this  time.  Activity  needs  to  be  retained 
to  clear  up  potential  problems. 

Investigator : 

One  way  communication  when  leader  clarified  partici¬ 
pant  manual.  Leader  asked  for  questions  or  problems  but 
received  few  responses  or  requests  for  additional  clarifica¬ 
tion.  Activity  appeared  worthwhile  and  important  for  later 
play. 

Activity  #2  (5-6) 

Leader : 


Classes  were  divided  into  groups  previously.  Fairly 
orderly  movement  into  groups  occurred.  Books  distributed 
efficienctly  and  quickly.  Some  confusion  when  drawing  role 
profile  and  picking  up  materials  on  part  of  residents. 

2nd  Developer: 

Division  of  class  into  groups  and  drawing  of  role 
profiles  adequate. 

Investigator : 

Appointment  to  roles  appeared  most  effective  and 
time  saving.  This  should  be  suggested  in  leader's  manual. 
Drawing  of  role  profiles  appeared  "chaotic"  momentarily. 

Have  only  one  member  of  group  draw  card  and  other  member 
get  information  booklets.  All  communities  were  represented 
by  the  random  draw. 

Activity  #2  (7)  +  3  (1/  2) 

Leader : 


Activity  took  longer  than  anticipated.  Extra  time 
required  stemmed  from  careless  reading.  Questionnaire  must 
be  simplified.  Too  many  good  students  going  too  slow. 

Maybe  specify  where  they  would  look  to  find  information. 

2nd  Developer: 

Activity  could  have  been  used  for  explicit  instruc¬ 
tions  using  overhead  or  blackboard.  Problem  areas  were: 

1.  Lack  of  directions  re  goals  for  administrators. 

2.  Confusion  about  interjecting  own  values  into  questions 
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where  information  not  provided.  3.  Lack  of  direction  re 
looking  for  information  in  both  role  profiles  and  information 
booklets . 

Investigator : 

Students  appeared  eager  to  get  into  game.  Need  for 
explanation  of  administrative  goal  setting  and  class 
presentations.  Leader-student  interaction  picked  up.  Need 
for  better  explanation  of  administrative  packages.  Extra 
time  was  required.  Students  permitted  to  prepare  outside  of 
class  at  their  request. 

Administrative  groups  ready  to  make  presentations 
before  resident  groups  had  completed  worksheets.  Noted  some 
unanticipated  interaction  between  administrative  group  and 
resident  groups  before  that  scheduled  activity.  Indicated 
eagerness  to  get  into  information  gathering. 

Activity  #3  (3) 

Leader : 


Some  administrators  had  difficulty  keeping  speeches 
under  two  minutes.  Talks  very  good.  Students  who  were 
nervous  spoke  quickly. 

2nd  Developer: 

Some  students  difficult  to  hear.  Need  to  impress 
students  with  importance  of  making  good  presentations. 
Possibly  important  to  have  more  vocal  people  in  class  acting 
as  administrators. 

Investigator : 

Resident  groups  very  attentive  to  speakers.  Wide 
variations  in  manner  of  presentation — varying  degree  of 
reliance  on  written  materials.  Only  one  utilized  map. 

Activity  #4  (1) 

Leader : 


Students  seemed  anxious  to  begin.  Some  prepared  to 
start.  A  couple  had  a  decision  made  before  they  began.  Some 
waiting  at  tables.  Maybe  5  minutes  is  too  long,  particularly 
after  first  round. 

2nd  Developer: 

Effective  because  of  previous  instructions  provided. 
Students  advised  to  concentrate  on  getting  information 
relevant  to  goals.  Less  time  needed. 
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Investigator : 

Leader  cautioned  about  time  limit  and  urged  to  get 
information  for  one  goal  each  time.  This  should  be  only  one 
suggestion  for  information  gathering.  Resident  groups 
decided  whether  to  split  up  or  work  together. 

Activity  #4  (2) 

Leader: 


Some  waiting  at  administrators'  tables.  Student 
administrators  didn't  seem  able  to  organize  themselves. 

2nd  Developer: 

Several  problems  evident.  1.  Administrative  groups 
unable  to  answer  questions  regarding  their  community. 

2.  Resident  groups  staying  entire  10  minutes  at  one  table 
asking  questions  related  to  many  goals. 

Investigator : 

Students  appeared  attracted  to  administrators  of 
same  sex.  One  administrative  group  chose  to  spend  all  its 
time  with  one  resident  group.  They  indicated  that  they 
preferred  a  few  citizens  totally  committed  to  the  community. 

Activity  #4  (3) 

Leader : 


Much  general  discussion.  Very  few  were  prepared  to 
make  decisions  at  this  point.  Most  had  just  picked  up 
general  information  rather  than  anything  specific. 

2nd  Developer: 

Failed  to  perceive  evidence  of  tentative  decision¬ 
making  although  there  may  have  been  some.  Perhaps  tentative 
decisions  should  be  written  down  on  a  card. 

Investigator : 

Students  made  no  observable  decisions,  possibly 
feeling  that  an  intermediate  decision  was  a  final  irrevers¬ 
ible  decision,  or  possibly  reserving  any  decision  until 
after  interaction  with  all  communities. 
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Activity  #4  (4) 
Leader : 


One  group  made  final  decision  here.  They  appeared 
not  to  have  considered  all  alternatives  but  my  questions 
didn't  seem  to  encourage  a  wider  consideration  of  the 
problem.  They  appeared  to  impose  their  values  over  those  of 
the  person  they  represented.  They  weren't  able  to  consider 
leaving  a  well  paid  job  because  of  an  ulcer.  This  was 
because  of  the  student^  personal  view  of  the  importance  of 
money. 

I  asked  students  to  re-read  goals  and  make  changes 
where  they  felt  necessary.  Students  appeared  to  take  this 
seriously  although  few  appeared  to  make  any  alterations. 

2nd  Developer: 

Heard  little  evidence  of  re-evaluation  of  goals. 

Might  be  a  good  idea  to  have  an  extra  sheet  attached  to  the 
questionnaire  for  writing  down  revised  goals.  Adminitrators 
did  not  appear  to  be  doing  anything. 

Investigator : 

Leader  clearly  indicated  the  importance  of  goal 
re-evaluation  in  real  life  and  encouraged  students  to  do 
this  with  their  simulation  goals.  Nevertheless,  while 
students  did  appear  to  deliberate  about  their  goals  for  a 
few  minutes  there  was  no  evidence  at  this  point  of  any 
changes  in  goals. 

Activity  #4  (5) 

Leader: 

This  lasted  longer  than  10  minutes  but  probably 
should  be  kept  shorter.  Groups  appeared  to  have  evaluated 
the  first  round  and  seemed  to  pick  out  problems  which  could 
be  remedied  on  subsequent  rounds  such  as  disorganization  and 
not  knowing  what  to  ask  and  not  having  enough  information. 
Most  students  indicated  that  they  hadn't  changed  their  goals, 
although  they  had  evlauated  them. 

2nd  Developer: 

Resident  groups  had  very  little  to  say  but  the 
administrative  groups  did  much  better  in  answering  question 
and  brought  out  points  mentioned  in  4-2 . 

Investigator : 

Students  should  be  encouraged  to  speak  loudly.  Only 
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one  group  had  visited  all  community  administrators  and  gave 
a  very  general  evaluation  of  their  performances.  Residents 
very  reluctant  to  speak.  Administrators  did  point  out 
issues  of  concern  and  problem  areas  for  them. 

Activity  #5  (1) 

Leader : 


Went  through  two  rounds  of  decision-making.  Groups 
appeared  to  know  what  they  were  doing.  Administrative  groups 
appeared  to  show  differences.  Some  appeared  to  attract  many 
people  for  inquiries,  others  very  few.  Zargay  administrators 
still  had  problems  finding  answers  to  questions.  Ross  came 
up  several  times.  Wastina  lost  people.  Few  came  to  ask 
questions.  Administrators  need  something  to  do  while 
residents  compute  points. 

Third  round  moved  quickly  as  students  knew  exactly 
what  they  needed. 

2nd  Developer: 

Rounds  of  decision-making  evidenced  more  interaction. 
Questions  and  answers  much  improved  over  previous  day. 
Students  moved  more  quickly  among  administrators.  Several 
problems:  related  to  unemployment,  when  to  draw  job  cards, 

several  groups  after  the  same  job,  resident  groups  not 
qualified  for  many  jobs,  difficulties  with  points  and 
formulas  which  must  be  thoroughly  revised. 

Activity  #5  (2) 

Leader : 


Students  had  good  insight  into  what  had  progressed. 
Some  students  indicated  that  when  computing  all  aspects,  a 
higher  salary  did  not  necessarily  indicate  that  you  came 
ahead.  Others  had  problems  when  they  drew  cards  indicating 
a  negative  situation.  Administrators  felt  that  they  had 
done  better. 

2nd  De ve loper : 

Students  vague,  uncertain  of  what  to  say.  There  may 
be  some  way  of  better  preparing  students  for  these  questions. 

Investigator : 

Administrators  appeared  to  take  a  much  more  active 
role  in  debriefing.  Resident  groups  were  often  inattentive. 
Noted  especially  one  group  which  had  bad  luck  at  the  draw 
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Activity  #5  (4) 
Leader: 


Formulas  caused  great  difficulty.  Satisfaction 
points  were  not  as  hard.  Worksheets  on  which  they  put 
points  need  revision.  Cut  out  lines  on  both  sides  and  leave 
in  appropriate  column  only.  We  must  develop  some  sort  of 
point  computation  for  administrators,  possibly  based  on 
their  personal  goals. 

2nd  Developer: 

Particularly  administrative  groups  had  problems  in 
computing  satisfaction  points.  A  lot  of  trouble  with 
formulas  all  of  which  must  be  thoroughly  revised. 

Investigator : 

Point  system  and  all  formulas  definitely  need 
revision  if  not  rejection  from  the  game  activities. 

Activity  #6  (2) 

Leader: 

Most  statements  were  very  brief  and  didn't  bring  out 
as  much  as  they  should  have.  I  think  we  should  set  a  form 
specifying  certain  information  that  must  be  included--e . g . , 

a.  read  last  paragraph  of  role — why  are  they  considering 
making  a  transition? 

b.  what  goals  set?  why? 

c.  what  actions  they  took  in  simulation? 

d.  what  outcomes? 

e.  satisfaction  points  they  gave  themselves. 

Otherwise  it  is  difficult  to  make  an  evaluation  of  whether 
they  played  the  simulation  well. 

2nd  Developer: 

Oral  presentations  indicated  uncertainty  over 
presentations.  However,  when  prompted  by  specific  questions 
they  gave  answers  indicating  good  awareness  of  what  was 
happening. 

Investigator : 

Groups  definitely  need  more  direction  in  how  to  make 
final  presentation.  Lack  of  questions  from  class  indicated 
that  many  did  not  really  understand  the  purpose  of  the 
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evaluation.  They  did  not  have  much  information  on  which  to 
evaluate. 

Activity  #6  (3) 

Leader : 


Evaluation  by  three  students  seemed  to  go  well.  All 
students  tended  to  give  bonuses.  I  don't  think  they  really 
got  into  the  critique  role.  They  should  have  been  able  to 
fault  some  for  the  way  they  played  the  role.  Need  to  work 
out  some  point  system  for  administrators  so  they  can  figure 
out  a  satisfaction  point  total. 

2nd  Developer: 

Evaluation  of  each  group  by  selected  students 
extremely  difficult  due  to  unsuitability  of  student 
questionnaire.  Repeated  next  day  with  revised  questionnaire — 
much  better  although  a  more  structured  approach  to  the  oral 
presentation  would  make  it  easier  for  evaluators. 

Investigator : 

Evaluation  of  groups  seemed  efficient,  although  one 
might  question  how  serious  the  groups  took  it.  All 
evaluations  did  not  seem  realistic.  Students  would  not 
subtract  points  from  a  group ' s  total  score  when  some  should 
have  had  points  taken  away  from  them.  Possibly  this  whole 
activity  needs  a  clearer  explanation.  Collection  and  tabula¬ 
tion  of  scores  by  student  group  was  most  efficient. 

Activity  #6  (4) 


Leader : 


General  class  discussion  seemed  a  bit  stale,  probably 
because  of  the  time  between  debriefing  and  the  final  round. 

I  brought  specific  observations  and  we  discussed  them  but 
few  students  brought  forward  any  "spontaneous"  reactions  or 
comments . 

2nd  Developer: 

Sorry,  I  don't  seem  to  have  any  mental  impressions 
of  the  effectiveness  or  non-effectiveness  of  this  activity. 

Investigator: 

Leader  appeared  very  organized  and  prepared  for  this 
final  debriefing  session.  Unfortunately  students  were  quite 
passive  when  it  came  to  offering  their  views.  They  may  have 
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forgotten  the  activities  of  five  days  before.  Success  of 
this  activity  greatly  depends  on  the  type  of  students 
involved.  Reactions  were  not  nearly  as  spontaneous  and  open 
as  they  had  been  during  the  initial  testing. 

Activity  #6  (5) 

Leader : 


I  felt  that  description  of  inquiry  process  was  too 
brief  and  hurried.  I  would  like  to  have  spent  more  time 
getting  students  to  show  what  they  did  and  see  if  they  could 
fit  it  to  the  model.  The  student  who  did  attempt  it  did  so 
adequately  but  I  think  we  could  have  pursued  the  model  more 
thoroughly.  Possibly  use  it  as  a  worksheet  and  find  out 
which  activities  occurred  according  to  the  model. 

A  little  pressured  at  the  end  because  of  the  need  to 
get  out  of  the  library. 

2nd  Developer : 

Use  of  transparencies  by  the  game  master  was  quite 
effective.  I  believe  that  students  were  able  to  understand 
what  the  inquiry  process  is  all  about. 

Investigator : 

Unfortunately  time  would  not  permit  extensive 
student  involvement  with  the  simulation  model.  The  student 
that  did  attempt  to  relate  his  activities  did  indeed 
indicate  that  his  group  had  used  the  inquiry  process  of  the 
decision-making  model  in  a  most  discernable  way.  I  think 
this  is  a  most  important  activity  with  which  to  conclude  the 
simulation  and  one  which  should  be  clearly  explained  in  a 
simulation  leader's  manual. 
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APPENDIX  B 


STUDENT  PARTICIPANTS'  QUESTIONNAIRE 


I  think  that  the  introduction  to  the  simulation  should 
have  been  (circle  one)  : 

a.  longer 

b.  shorter 

c.  the  length  it  was 

I  think  that  the  following  things  should  be  added  to  the 
game : 

a. 


b. 


c . 


d. 


(Add  more  comments  on  the  back  of  this  sheet  if  you  need 
to) 

I  think  that  the  following  things  should  be  taken  out  of 
the  game : 

a. 


b. 


c . 


d. 


I  found  the  following  things  difficult  to  understand: 
(check  those  that  you  found  difficult  and  write  comments 
if  you  wish) 

Comments 


the  introduction  (what  a  simulation  is) 
the  background  information 

the  description  of  the  problem  of  transition 
in  the  world  today 
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_  the  description  of  the  systems  in  a  community 

_  the  rules 

_  the  information  on  my  community 

_  the  role  description 

_  the  Work  Sheet 

_  the  activity  of  setting  goals 

_  the  information  which  the  resident  groups 

wanted  when  they  came  to  me  for  information 
(Administrative  groups  only  check  this  one) 

_  the  information  which  the  administrative 

groups  had  when  I  went  to  them  for  information 
(Resident  groups  only  check  this  one) 

_  making  a  final  decision 

_  the  formulas 

_  the  credit  points 

_  the  evaluation  of  my  group  by  other  students. 

I  found  the  following  things  easy  to  understand: 

(check  those  that  you  found  easy  and  write  comments  if 
your  wish) 


the  introduction 

the  background  information 

the  description  of  the  problem  of  transition 
in  the  world  today 

the  description  of  the  systems  in  a  community 
the  rules 

the  information  on  my  community 
my  role  description 
the  Work  Sheet 

the  activity  of  setting  goals 
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_  the  information  which  the  resident  groups 

wanted  (For  Administrators  only) 

_  the  information  which  the  administrative 

groups  had  (For  Resident  groups  only) 

_  making  a  final  decision 

_  the  formulas 

_  the  credit  points 

_  the  satisfaction  points 

_  the  evaluation  of  my  group  by  other  students. 

I  found  that  my  role  which  was _ in  the 

community  of  _ _ ,  (difficult)  (easy)  to 

understand  because 


I  think  I  played  my  role  (well)  (poorly)  because 


I  think  people  playing  this  role  need  (more)  (less) 
information  because 


I  think  that  my  role  was  (important)  (unimportant)  to 
the  success  of  our  group  because: 


I  think  that  our  group  has  an  (important)  (unimportant) 
role  in  this  game  because 


I  think  that  this  game  is  (too  difficult)  (too  easy) 
(just  right)  for  grade  nine  pupils  because 


(vlrro  e*o:Jfix*8i0iin£>A  20I) 


' 


-  ^ 


. 


■ 


■ 


\ 


...  :  * : .  .  t  <  ' 


136 


12.  I  was  (happy)  (not  happy)  with  my  group's  final  decision 
in  the  game  because 


13.  I  think  that  I  would  play  this  game  (better)  (the  same) 
(worse)  if  I  were  to  play  it  again  because 


14.  I  found  this  game  (interesting)  (uninteresting)  because 


If  you  have  any  suggestions  for  improving  the  game 
please  feel  free  to  add  them  below. 

Suggestions 
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APPENDIX  C 


TIME  LINE:  SIMULATION  ACTIVITIES 

Activity  #1:  Pre-Run  Operation:  Introduction 

1.  Game  Master  explains  briefly  the  development  of 
Transition  as  associated  with  Project  SURT. 

(Videotape) 

2.  Game  Master  indicates  the  importance  of  the  role  of 
students  in  the  formative  evaluation  of  Transition. 

(Videotape) 

3.  Game  Master  explains  what  a  simulation  is  via 
comparison  of  simulation  with  an  athletic  game.  Use 
of  transparencies  and  overhead  projector. (Videotape) 

Total  Time  for  Activity  #1:  10-15  minutes 


Activity  #2:  Introduction  to  the  simulation  Transition. 

1.  Game  Master  describes  the  provinces  and  communities 

being  simulated  in  Transition.  Use  of  wall  map  and 
transparencies.  (Videotape) 

2.  Game  Master  explains  the  types  of  groups,  roles  and 
generally  what  is  required  of  simulation  participants. 

10  minutes  (Videotape) 

3.  Game  Master  passes  out  participant  manuals  and 
participants  read  manuals. 

15  minutes 


4.  Game  Master  clarifies  information  in  manuals. 

5  minutes  (Videotape) 

5.  Game  Master  divides  class  into  groups  and  designates 
areas  of  the  classroom  for  each  group. 

6.  Participants  draw  role  profiles  and  assume  class 
positions. 

10  minutes 

7.  Groups  each  read  their  information  packs. 

20  minutes 

Total  Time  for  Activity  #2:  55  minutes 

Activity  #3:  Worksheet  and  Goal  Setting  Decisions 


1.  Resident  Groups  complete  worksheet  and  set  goals  and 
number  specific  goals  in  the  order  they  will  seek  to 
achieve  them.  (Audiotape) 
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2.  Administrative  groups  will  set  goals  and  prepare 

short  statements  for  presentation.  (Audiotape) 

25  minutes 

3.  Administrative  Groups  present  statements  to  residents. 

(Videotape) 

5  minutes 

Total  Time  for  Activity  #3:  30  minutes 

Activity  #4:  Goal  Achievement  Decisions 

1.  Resident  Groups  prepare  for  information  gathering. 

5  minutes 

2.  Resident  Groups  gather  information  via  interaction 

with  administrative  groups.  (Videotape) 

10  minute  time  limit 

3.  Resident  Groups  decide  what  they  will  do  and  then  go 
to  the  drawing  board  and  select  a  draw  card. 

5  minutes  (Videotape) 

4.  Resident  Groups  re-evaluate  goals  and  decide  what  to 

do  in  the  new  situation  presented  by  the  draw  card. 

5-10  minutes  (Audiotape) 

5.  Resident  Groups  compute  points. 

5  minutes. 

6.  Intermediate  class  debriefing  session. 

Game  Master  asks:  general  questions:  what  happened? 
Why?  What  did  you  want  to  happen?  What  re-evalua- 
tion  took  place?  What  influence  did  people  have  on 
your  decision  as  opposed  to  factual  information? 

(Videotape) 

Game  Master  asks  administrative  groups:  what  did 
you  observe?  What  kind  of  questions  did  resident 
groups  ask?  How  did  you  budget  your  time? 

(Videotape) 

10  minutes 

Total  Time  for  Activity  #4:  45  minutes 

Activity  #5:  Whole  Activity  #4  Process  Repeated. 

1.  Decisions  made  regarding  living  accommodations,  job, 

cost  of  moving,  cost  of  living,  recreation,  government, 
education,  transportation.  (Videotape) 

2.  Debriefing  after  every  other  round  of  decision¬ 
making.  (Videotape) 

3.  Game  ends  when  goals  are  achieved. 

4.  Groups  compute  total  satisfaction  points. 

Time  Variance 

Maximum  Total  Time:  80  minutes 
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Activity  #6:  Final  Debriefing  Session. 


1.  Each  resident  and  administrative  group  prepare 
statements  for  class  presentation: 

What  were  our  goals?  What  was  our  situation  to 
begin  with?  What  decisions  did  we  make?  What 
satisfaction  points  did  we  give  ourselves  and  why? 

5  minutes  (Videotape) 

2.  Each  group  makes  class  presentation.  (Videotape) 

20  minutes 

3.  Evaluation  of  each  group  by  selected  students. 

10  minutes 


4. 


5. 


General  class  discussion  of  what  occurred  during  the 
game.  (Videotape) 


Game  Master  describes 
the  game. 


the  inquiry  process  used  in 

10  minutes  (Videotape) 


Total  Time  for  Activity  #6:  45  minutes 


Transition  Total  Time:  4  hours  30  minutes 

Activity  #7:  Post- Run  Operation 

1.  Six  students  from  various  groups  represented  are 

interviewed.  (Audiotape) 

Remaining  simulation  participants  complete  a  question¬ 
naire  re  their  simulation  experience. 

30  minutes. 
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APPENDIX  D 

PHOTOGRAPHS  OF  SIMULATION  TRANSACTIONS 
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PICTURE 

1  Simulation  leader  introduces  Transition. 

2  Participant  groups  read  materials  and 
engage  in  designative  inquiry. 

3  Administrator  makes  class  presentation 
to  promote  his  community. 

4  Participant  groups  engage  in  appraisive 
inquiry  as  they  decide  on  their  goals  for 
the  simulation. 


141 


/ 

. 


■ 

.  • 


142 


PICTURE  3 


PICTURE  4 
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PICTURE 
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Resident  groups  interact  with  administrators 
and  move  toward  prescriptive  inquiry  and 
final  decision-making. 


8  Participant  groups  re-evaluate  goals. 
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PICTURE  7 


PICTURE  8 


PICTURE 

9  Administrators  prepare  for  additional  rounds 
of  interaction. 

10  Simulation  leader  conducts  an  intermediate 
debriefing  session. 

11  Participants  compute  credit  and  satisfaction 
points  after  decision-making. 

12  Particpants  engage  in  final  debriefing. 
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PICTURE  10 


PICTURE  9 


PICTURE  12 
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